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ABSTRACT 



Annual Reviews are the most ambitious and highly organized members 
of the class of communication tools that attempt to summarize and 
critically evaluate widespread activities. The basic objective of 
this study, supported by the U.S. Office of Education, System Devel- 
opment Corporation, and Encyclopaedia Britannica, was to examine, 
apparently for the first time in communications research, the impact 
on professional communication of an annual review publication. A 
sample of users and non-users of Volume 1 of the Annual Review of 
Information Science and Technology was identified, and question- 
naires were sent to nearly 3 >000 of them. The respondents were 
divided into three groups, for analysis: those who had used the 

Annual Review, those who had only seen it or heard of it, and those 
who had not yet seen it or heard of it. Users of the Annual Review 
differed from non-users in a Ox ways, particularly on mea- 

sures of professional activity level, where they were uniformly 
higher than the non-users. They reported a suiprising variety of 
valuable uses; current awareness in one’s own area of interest; 
current awareness in peripheral areas; learning in new areas; 
checking on particular projects, activities, ideas, or individuals; 
serving as a classroom text; and, simply, browsing. Although im- 
pact studies generally require study over time, the impact of the 
Annual Review is already partially evident from the tendency of 
users to re-examine cited literature again, to seek new cited 
literature, and to seek contact with the authors of cited litera- 
ture. Both the questionnaire and severed informal interview 
sessions sought to elicit suggestions for improvement of the 
series. Some inprovements have already been made or are being 
considered. Others, like making the Annual Review more tutorial 
and less technical, have not, because they involve the serious 
risk of making it less useful to the professionals whose activities 
are most likely to accelerate progress in information science. This 
study suggests that annual reviews have great potential for advanc- 
ing many fields of inquiry and should, where possible, be encouraged 
and supported, preferably without demanding so precipitous a rate of 
growth toward self-support that the dilution in quality is ultimate- 
ly self-defeating to the sponsoring organization. 
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1. INTRODUCTION AND SUMMA RY 

1.1 PURPOSE AND STRUCTURE OF THIS REPORT 

This is the final report on the work of the Annual Review Impact Study at 
System Development Corporation. Tt.ie purpose of the study has been to develop 
a better understanding, based on empirical evidence, of the functions of 
annual review publications in professional communication, and particularly of 
Volume 1 of the Annual Review of Information Science and Technology ! and to 
discover ways in which that particular review can be improved. 

The study was carried out in two phases. The first phase, from June, 1967 
through March, 1968, comprised the project planning, the development of survey 
tools, and the conduct of a questionnaire and interview survey. This phase of 
the work was supported by System Development Corporation, with some assistance 
from Encyclopaedia Britannica, the current publisher of the Annual Review. 

The second phase of the work comprised data reduction, data analysis, the in- 
terpretation of results, and the preparation of this report. This phase of 
the work was supported largely by the Library and Information Branch, Bureau 
of Research, U,S, Office of Education. 

This final report describes all of the work of the project and supersedes any 
prior progress reports, published or unpublished. Sufficient detail is pro- 
vided, both in the text and in the appendices, to permit adequate understand- 
ing and technical appraised by researchers and others, and edso to permit 
replication of all or part of the study. In the event that interested re- 
searchers require information beyond that contained in this report, members 
of the project staff will be pleased to respond to specific inquiries. 

The final report consists of six sections, plus appendices. This first sec- 
tion includes a summary of the entire study. Section 2 describes the back- 
ground for the study and indicates the purposes to be served in undertaking 
the 'study. Section 3 describes the design and execution of the survey, which 
was the major work in the study. Section k describes the data analysis proce- 
dures. Section 5 presents and discusses the results of the survey. Section 6 
presents the conclusions and the iisplications of the study for professional 
communication, for survey miethodology, and for the Annual Review. 



1.2 SUMMARY 

1.2.1 Issues and Goals in Assessing Professional Comniunl cation Techniques 

As the volume and variety of information needs and uses have increased, many 
new information- communication tools have been developed and placed into opera^ 
tion. Partly because they can be very costly, they must be carefully evalu- 
ated, not only to assess their individual utility (effectiveness) but also to 
discern their effect (or "impact") on the use of other communication tools. 
Such evaluation is necessary to adapt present communication institutions and 
forms to changing times and to guide the development and financial support of 
new institutions and forms. 
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Annual reviews are the most ambitious and highly organized members of the 
class of communication tools that attempt to summarize and critically evaluate 
widespread activities. While there are many annual review series in existence, 
and more are being proposed and planned, there have been no empirical studies 
of their impact on professional communication. The establishment in I966 of 
the Annual Review of Information Science and Technology , under the auspices of 
the American Society for Information Science (then the American Documentation 
Institute) and System Development Corporation, provided a unique opportunity 
to undertake such a study, within a field that is itself vitally concerned 
with communication. Accordingly, the study was initiated by SDC in July, 1967* 
The two primary goals of the study were (l) to assess the impact of Volume 1 
of the Annual Review on professional communication within the information 
science community, and (2) to obtain information regarding means of in^roving 
the Annual Review in subsequent volumes. It was also hoped that the study 
might provide some preliminary guidance to funding agencies, professional 
societies, and other groups interested in supporting annual review publica- 
tions. 



1.2.2 Design and Execution of the Survey 

The main work in the study was that of conducting a survey of users and non- 
users of the Annual Review. This involved three main tasks: 

(1) Identifying a suitable sau5>le of users and non-users. 

(2) Determining, by means of questionnaires and interviews, the 
attitudes and practices of these groups in relation to infor- 
mation sources, including the Annual Review. 

(3) Analyzing and interpreting the results. 

It was decided that using a mail questionnaire would be the most efficient and 
least expensive way to supply the bulk of the data. Personal interviews were 
held to amplify and verify the questionnaire data and to permit deeper explora- 
tion of particular topics. 

To identify the users, business reply postcards were sent to approximately 
10,000 members of four professional organizations: the American Society for 

Information Science, the Special Libraries Association, the Association for 
Computational Linguistics, and the Special Interest Group on Information 
Retrieval of the Association for Confuting Machinery. The postcard asked 
whether the person had seen, heard of, or used the Annual Review, and which 
one of eight job categories accounted for the major portion of his profession- 
al activities. The first question divided respondents into three groups: 
those who had used the Annual Review (Users), those who had only seen it or 
heard of it (Seen/Heard group), and those who had not yet seen it or heard 
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of it (Nbt-Seen/Heard group). The two groups of non-users were included in 
the survey for the purpose of comparing literature^use practices and other 
attributes, such as education, eaperience, place of employment, etc. 

Nearly 3,700 postcards were retiurned, adn»st equally divided among the three 
groups. From the returns, a sample of about 3,000 was. selected for the ques- 
tionnaire mailing. Separate questionnaires were developed for the three 
groups, the User questionnaire being the most detailed. The questionnaires 
had 12 questions in common that were related to information- seeking, so that 
the three groups could be conpared on demographic attributes. 

Each version of the questionnaire was pre-tested and revised, using ASIS and 
SLA members in the Los Angeles area. Most of the questions were pre-stzhic- 
tured (i.e., they provided a list of choices) to make the coding and data 
analysis easier. Some open-ended questions were included to stimulate sugges- 
tions for improving the Annual Review and to provide for responses that could 
not he anticipated when the questionnaire was written. 

An original and one follow-up Mailing were made, yielding an overall percentage 
return of 70^. The final data set contained 2,012 usable questionnaires. In- 
terview data to supplement the questionnaires were obtained at the October, 
1967, Annual Meeting of the ASIS in New York, at a special meeting in Chicago 
that same month, and at the I968 Annual Meeting of the ASIS, In Columbus, Ohio. 
These sessions were used tp e^qplore particularly Intriguing questionnaire 
responses and to obtain suggestions for the future improvement of the Annual 
Review. 



1.2.3 Data Analysis 

The primary goal of the data analysis was to achieve an accurate and meaningful 
description and interpretation of the 2,012 replies to the questionnaire. For 
most of the data, the primary analysis technique was to construct frequency 
tables by comparing (cross-tabulating) criterion variables (such as number of 
Annual Review chapters read) against classification variables (such as primary 
job activity). In several instances, where a single formattable question was 
not adequate to reflect a certain attltudlnal or behavioral phenomenon, the 
responses to a number of different questions were combined to form a single 
index, and the distribution of responses for the index was taken to represent 
the desired measure. Four such indexes were developed: (l) the Prbfessional 

Organization Index, r^resenting extent of organizational membership, 

(2) Journals Read Regularly, representing extent of journal readership, 

(3) Diversity of Authorship, representing professional publications activity, 
and (4) the Read/Skim Index, representing the extent to which the Annual 
Review was actually used. 
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The coding procedure for nonformat table data was to examine, verbatim, all 
responses to each question, and then, by looking at the verba ta, to derive for 
each question a set of categories, based on some conceptual scheme, that ade- 
quately represented the content of the replies. The data gathered through 
Individual and group Interviews and discussions were, of course, nonformattable 
and required this kind of qualitative, rather than quantitative, analysis. 

1.2.4 Results 

1.2. 4.1 Characteristics of Annual Review Users 

The largest numbers of Annual Review Users were employed In Industrial or 
business firms, with "management or administration" listed most frequently as 
an Important Job activity. More than one~thlrd of the Users reported spending 
time In reference or other library work. The two non~User groups tended to 
have the same occupational settings and types of work, but there was a greater 
preponderance of Industrial over university settings. The Seen/Heard group 
were engaged, more than Users and Not-Seen/Heards, In library reference and 
other library work. It was difficult to characterize the Not-Seen/Heard group; 
they were seemingly the most research oriented and teaching oriented of the 
three groups, but they were also more actively engaged In programming than 
the other two groups. 

The User group differed from both of the non~User groups In having proportion- 
ately more professionally trained librarians, although more non-Users than 
Users reported actually spending on-the-job time In library work. The Not- 
Seen/Heard group appeared to be rather heterogeneous In educational and pro- 
fessional background, and may have Included many newcomers to the Information 
science field. Twice as many persons In the Not-Seen/Heard group had earned 
degrees In llgulstlcs, mathematics, and computer science as had persons In the 
other two groups; a possible explanation for their being non-Users Is that 
their Information channels are the computer literature — not the Information 
science literature that brings the Annual Review to the attention of Users. 

More than half of each of the groups reported having worked In the field of 
Information science for 10 years or less (In spite of the fact that the defini- 
tion of Information science used In the survey was purposely made very broad) . 
The respondents were fairly young, professionally, or they considered Informa- 
tion science as a relatively new field of activity, or both. Users showed the 
highest levels of professional activity of the three groups. They reported 
reading more journals regularly, authoring more kinds of papers, attending more 
professional meetings per year, and presenting more papers at those meetings. 

Users of the Annual Review showed no tendency to use It as a substitute for 
reading the primary literature: persons who read and skimmed the book exten- 

sively also read journals extensively. (This finding was In agreement with 
the findings of readership surveys In general. In which It Is usually found 
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that extensive reading of one type of material is highly correlated with 
extensive reading of other ^types of material.) 

The relative value placed on various information source differed. All three 
groups reported relying heavily on Joumeds and periodical literature, but, in 
comparison with the other two groups. Users tended to place greater value on 
out-of‘-house colleagues, professioncil meetings, technical reports, and pre- 
prints— in a sense, the informal information channels. Both the Seen/Heard 
and Not-Seen/Heard groups placed somewhat more value on books than Annual 
Review users did. 

An incidental characteristic of the sample was the higher questionnaire 
response rate for Not-Seen/Heords over Seen/Heards, and, in tura, of Seen/ 

Heards over Users. The percentages of returned questionnaires appear to be 
inversely related to the length of the questionnaires. Another incidental 
characteristic was that about 60 % of the respondents were males, this finding 
perhaps reflecting the membership characteristics of the organizations polled. 

The highest proportion of males was found among the Users and the lowest in 
the Seen/Heard group, perhaps because the highest percentage of practicing 
librarians is in the Seen/Heeurd group, and the majority of practicing librarians 
are women. 

1.2. 4. 2 Acquisition of Annual Review Copies 

Almost 60 % of those in User group used a copy of the Annual Review that was 
more or less permanently available to them after they had put forth the initial 
effort to obtain it. This may attest to the need to have a copy easily avail- 
able, for more effective use. University professors and staff tended to pur- 
chase the Annual Review, \^ile employees of government and industiy tended to 
rely on access to company-p\u:chased copies. Twenty-five percent of the Users 
borrowed a copy through a library. 

Abc^ut half of the Users first heard of the Annual Review through an ASIS channel : 
announcements sent- through the mail or made at the 1966 Annual Meeting. In 
contrast, only l4/{ of the Seen/Heard respondents heard of the book through ASIS 
information channels. This group, as a whole, did not appear to be greatly 
interested in information science, and fewer members of this group were ASIS 
members. 

1.2.4. 3 Extent of Interest and Readership 

The Annual Review was intended to provide good coverage of the range of the 
interests of its ASIS audience. This intent was reflected in the diverse sub- 
ject matter of the 12 chapters. Users, the majority of whom are ASIS members, 
cited an average of five chapters that they considered to be within their fields 
of interest; Seen/Heard respondents, fewer of whom are ASIS members, cited an 
average of four. 
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The Users' professional interest.^ appeared to be different from those of the 
Seen/Heard group. Users named the chapters that dealt with library automation 
and with information needs and uses as the two most closely related to their 
interests. The Seen/Heards placed thesiii areas third and ninth, respectively, 
listing file organization and infoi-mation centers and services as their areas 
of primary interest. Both groups placed automated language processing at the 
bottom of their interest list. The Users reported reading approximately four 
chapters in detail and skimming another four. Since some Users had not had 
Volume 1 for very long before answering the questionnaire, the total figure of 
eight chapters must be assumed to be an underestimate of the extent of reader- 

ship. 



1.2. 4. 4 Purposes Served by the Annual Review 

To find out how well Volume 1 of the Annual Review served reference, current 
awareness, and other important information functions, both a structured and a 
"critical incident" question were included on the Users' questionnaire. 

Although the responses were not strictly comparable , they gave similar indi- 
cations of the primary uses of the Annual Review: for keeping up with current 

work in one's areas of interest and in peripheral areas of interest, for 
checking on particular projects or ideas, for learning about areas not within 
one's current professional specialty, and for reading the original literature 
more selectively. Nearly a third of the Users also reported finding the 
Review moderately useful or very useful for identifying areas of information 
science that require further research. That use is particularly interesting, 
since there are probably relatively few readers whose work calls upon them to 
engage in such Identification. 

It is noteworthy that the use of the Annual Review for specific reference was 
not as prominent as its use for current awareness end learning, but it would 
have been unlikely for the Annual Review to be of great usefulness in specific 
reference work after the publication of only one volume. The Seen/Heard group 
was asked how they might, in the future, make use of the Annual Review. They, 
like the Users, emphasized current awareness, rather than reference. The 
figures obtained probably underestimate the eventual reference use of the 
series. 

Perhaps the most interesting finding regarding uses of the Annual Review is 
that there are so many different ones, including low-frequency uses such as a 
classroom text and an aid in preparing speeches. A comparison of usefulness 
ratings with extent of readership showed that the higher the usefulness indi- 
cated for each of the purposes, the higher the readership. No significant 
difference in purposes of use were found among different groups of Users whose 
typical information-seeking activities were print oriented, colleague oriented, 

and so on. 
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1.2iU'.5 Correlates of Previous Contact with Review-Cited Literature 

All biit 9^ of the Users reported at least "some" prior contact with puhlica- 
tiohs in their areas of' interest that were cited in the Annual Review* A 
third of the Users reported reading "quite a few" of these publications, and 
of them said that they had read most of them. Analysis showed that the 
extent of previous contact was significantly related to only one purpose of 
reading the Annual Review: maintaining current awareness in one's own area 

of interest, ihose Users with high previous contact tended to find the Annual 
Review much less useful for Current awareness than those with low previous 
contact, although this tendency decreased as the number of chapters read 
increased. That is, the more chapters were read, the more useful was the. 
Annual Review for current awareness in one's own area — even with high previous 
contact. Presumably, it is the critique that is important, rather than the 
alerting function. One may wonder how these functions will change in impor- 
tance as the literature in information science expands. If the authors are 
forced to refer only briefly to many studies, readers may need to have exten- 
sive previous contact with the publications cited in order to reap some of the 
benefits of the Annual Review. 

While the worth and function of high previous contact with cited publications 
is not yet fully evident j it is Clear that previous contact with the litera- 
ture did not deter^ extensive reading and skimming. It was found that Users ' 
with little previous literature contact also tended to be low on extent of 
readership, as measured by the Read/Skim Index. However, Users with high 
previous contact were equally likely to be low or high on the Read/Skim Index. 
There was a positive relationship between previous contact and various kinds of 
information activities, such as attendance at professional meetings and presen- 
tation of papers at these meetings. Perhaps a general factor relating to high 
professional activity level underlies this relationship. 

1.2. 4 ^6 Relationship of Annual Review Readership to Other User Characteristics 

A number Of relationships were found between the extent of readership of the 
Annued Review, as measured by the Read/Skim Index, and other characteristics. 

A cross-tabulation of this Index against the "professional age" variable showed 
that Users who had worked in the information science field for less than 10 
years tended to use the book less than those who had been in the field longer. 
The most likely explanation for this is that newcomers do not yet have suffi- 
cient breadth and depth of experience to need a tool like the Annual Review or 
to use it effectively. This possibility is supported by the finding that the 
majority of Users with more e3q>erience who found the Review "very useful" for 
current awareness were also high oh the Read/Skim Index. 

There is also some relationship betweep readership and primary job activities. 
Persons engaged in library reference work tended to use the Annual Review less 
than average. Consultants, on the other hand, tended to be heavy users. This 
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is not surprising, since consultfiuats obviously have to read extensively to 
maintain their competence in their field. Nor is it surprising that a positive 
correlation was found between extent of readership and rated importance of the 
Annual Review as an information source. The more the Annual Review is read, 
the more important it becomes to the readers. 

1.2. 4. 7 Impact of the Annual Review 

To evaluate the impact of a particular information technique or tool, it is 
necessary to obtain data over a period of time. Since the present survey 
involved data obtained at only one point in time, a true measure of Impact is 
not yet available. Nevertheless, some conclusions can be drawn on the basis 
of some of the questionnaire items. 

More than 90 % of the Users reported a,t least "some” prior contact with the 
literature cited in the Annual Review. Nevertheless, more than half of those 
Users were led to reexamine the literature. Thirty percent of those who 
examined the literature again did so to "review details ' ; twenty percent said 
that they examined a publication again to reevaluate it in the light of an 
Annual Review author's comments. 

Over half of the Users tried to obtain Annual Review-cited literature that they 
had not seen before, indicating considerable acceptance of the volume as an 
authoritative guide to valuable literature. Additional corroboration of the 
stimulative effect of the Annual Review was the fact that lOJt of the Users 
reported attempting to contact cited authors after reading the Annual Review. 
Forty-five percent of the Users reported that the Annual Review suggested to 
them specific ideas for future research projects or studies. 

Of some interest is the fact that persons who had had high previous contact 
with cited publications had a greater tendency to reexamine publications and 
to contact mentioned authors than did those who had had low previous contact. 
This may be more evidence of the level-of-professional-activity dimension 
mentioned earlier. There is clear evidence that as the extent of readership 
increases, so does every form of post-reading behavior. This strong correla- 
tion of reading with important post-reading behavior suggests that the Annual 
Review, even its first volume, had fulfilled one of its goals: to stimulate 

p*'oductive activity in information science. 

1.2. 4. 8 User Suggestions for Improvement 

Users had a number of opportunities to offer suggestions for changes and 
improvement in the Review. They were asked whether they would prefer that one 
chapter be published each month instead of collecting the chapters in book 
form annually; the annual form was preferred by two to one. Explanations 
indicated that the annual volume is preferred because it surveys the entire 
state of the art, is more permanent, and has an index. 
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Suggestions for improvement covered style of writing, contents, organization, 
information access tools, and mechanics of production. It was suggested that 
authors give more detailed information — e.g., about the results of studies — 
and that they be more critical and evaluative. There were a number of sug- 
gestions for better organization of the chapters and more cross-comparison 
among them. Suggestions were given for adding~and, in a few instances, 
deleting — specific chapters. It was also suggested that the publication cycle 
be changed so that the book would be available before the ASIS convention 
(normally in October). 

Some of the ideas expressed in these suggestions have already been adopted, 
either in Volume 2, which became available at approximately the time the 

survey was underway, in Volume 3, published in September, 1968, or in Volume 4, 

now in preparation. The publication time has been shortened; Volume 3 covered 
the full calendar year 1967 but was published in September, 1968, a full month 

before the ASIS convention. 

A "Feedback Forum" held with the Editor of the Annual Review and staff members 
at the 1968 ASIS Annual Meeting developed additional interesting suggestions. 

It was suggested, for example, that the inforuiation science field did not 
require an annual review, since there was so little "real progress." Other 
suggestions that drew more audience support were for better coverage of the 
secondary services and the addition of a single overview chapter for each 
volume . 

1.2.5 Discussion 

1.2. 5.1 Role and Impact of the Annual Review 

The Annual Review of Information Science and Technology serves many different 
and important purposes. It has demonstrated, even in its first year, an impact 
on professional communication. It does not appear to reduce the use of other 
information channels but may, in fact, enhance their use. Information channels 
tend to reinforce each other, and the effects are strongest for channels, such 
as reviews, that point toward other information channels. 

The possibility exists that the questionnaire technique used in the study 
inflated the measures of impact, by making the respondents conscious of being 
studied. Any such inflation would probably be offset, however, by the rela^ 
tively brief exposure that the Annual Review has had for many users. Responses 
based on such brief exposure undoubtedly underestimate the potential usefulness 
of the series, particularly since the existence (in 1966) of only one volume 
limited its use for reference purposes. The impact of the Annual Review is 
probably much greater now than it was at the time of the survey, not only for 
these reasons indicated, but also because various improvements have continued 
to be made. Some suggested improvements, e.g., adopting a more tutorial and 
less technical orientation, have not been made because, even though they could 
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Increase audience appeal and sales, they would make the Annual Review less 
useful to the vary professionals whose activities are most likely to help 
accelerate the growth and advancement of Information science. 

1.2. 5. 2 Methodological Implications of the Study 

The present study was concerned with both Impact and effectiveness. One 
needs to consider both because they are Interrelated and because the distinc- 
tive characteristics and effectiveness of a communication tool are determining 
factors with respect to Its Impact. 

The response rate on the survey was surprisingly high, and the data are very 
rich and provide a good baseline for future studies of the Annual Review. 

Also gratlfyingly useful was the experimental design, which permitted many com- 
parisons between users and non-users of the Annual Review. 

Since this is the first extensive empirical study of an annual review publica- 
tion, it is highly desirable to schedule follow-up studies, perhaps beginning 
in the Fall of 1969. The questionnaire technique may be useful to continue, 

although one would certainly make some changes in particular Items, to make the 

questionnaire a more sensitive Instrument for our purposes. It would also be 
desirable, in follow-up work, to adopt, where possible, somewhat less obstruslve 
measurement procedures, to avoid conscious or unconscious response bias. 

1.2.5. 3 Implications for Sponsorship of Annual Reviews 

This study suggests that annual reviews may have enormous potential for ad- 
vancing many fields of inquiry and should, where possible, be encouraged and 
supported. Although It is not easy to determine the clrcuTAStances under which 
a "field" exists and Is ready for an annual review publication, one should keep 
In mlng that an annual review may Itself be able to play a role In accelerating 
the cohesive development of a field. Some areas, like computational linguistics, 
do not yet seem ready; other areas, such as education, seem over-ready for 
annual reviews. 

Should the Federal Government support the development of annual review publica- 
tions? The answer would certainly seem to be "yes" for fields in which the 
Government has an Important. stake, since annual reviews demonstrably enhance 
professional communication and, therefore, technical progress. Care must be 
taken, however, by sponsoring organizations — Federal or other — not to expect of 
demand too precipitous a rate of movement toward self-support. If such pressure 
leads to quality-diluting shortcuts or low technical standards. In the prepara- 
tion of reviews, this will ultimately be slef-defeatlng to the sponsoring 
organization . 

Whatever the kind and extent of annual review support. It Is Important to con- 
tinue the empirical study of such publications, to help provide better-informed 

judgments about current and promising communication tools. 
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2 . AS3iI S a^SM] OF !PR0FELSSI0NAL CDMMUNIOATIIiON; ISSUES AND GOALS 



2.1 ASSESSING NEW CONMUNICAEEON 'EECHNIQUES 

It is veil recognized that the large-scale transfer of scientific and 
technical information has becoifle progressively more ramiform and complex 
as the variety and volume of information needs have increased and a 
concomitant variety of comunmi cation channels and techniques have developed. 
Science and technology have generated new Information at such an accelerated 
pace that scientists and technicians have foxind it increasingly difficult 
to locate, store, retrieve, and disseminate information effectively. 
Recognition of this difficulty has led to many experimental innovations, 
both in organization and techniques, as older approewhes to information 
transfer are found to be inadequate. 

Ihe problem has gained national attention, in the Congress, in the executive 
branch of the Federal Govermoent,. and in the private sector. Over the past 
twelve years, mou’e than twenty major studies have focused on problems of 

scientific communication# Qhe concern is evident in many quarters, as 
witness the emergence of such organizations as EDUCOM, COSAIEI (Committee on 
Scientific and Ofechnical Information), and SATCOM (Scientific and Otechnlcal 
Communication) . 

Qhe concern is also evident in the extensive development of new facilities 
and servJ.ces such as information analysis centers, document clearinghouses, 
and new hinds of indexing and abstracting services. It is also reflected 
in the w:Lllingness of several kinds of organizations, both public and 
private, to s\;pport basic le search and advanced development directed toward 
the invention of more effective transfer mechanisms. 



Some of the suggested solutions to this complex problem have Included 
application of conputer technology, technical and organizational changes in 
primary and secondary publication mechanisms, changes in the organization 
of national policy-making groups, and massive application of Federal 
resources. Many of these suggestions are already being tried, with varying 
degrees of success. New Isechniques in Information retrieval and selective - 
dissemination-of-information have been experimented wj.th in many disciplines 
and areas of scientific and technical work. Reprography and printing 
technology are both developing rapidly and are having many Inportant effects 
on the rate at which innovations in information transfer can be developed 
and inplemented. Finally, the information problem is being alleviated by 
data conpilations and analyses, literature reviews, and state-of-the-art 
studies. 

Itiese several innovations are costly and need to evaluated, particularly 
in situations where they are conpeting for the same support funds. Ohey 
need to be assessed not only in terms of tlieir individual utility and 
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effectiveness, but. also wj.th respect to their interaction and mutual inpact 
as communication channels and devlt^s. Evaluating Information systems, and 
services and media., however, is still difficult because evaluation 
techniques are not yet grounded upon an adequate body of theory and practice, 
and hi^ly reliable and replicable evaluation techniques are yet to be 
developed. Nevertheless, some of the methods and tools adapted i^om the- 
behavioral sciences can be useful, both in developing a more viable 
evaluation technology, (md in Icarrlng more about communication techniques. 
One such method or approach will be discuseied here, in connection with the 
project to ausuess the? initial iiipact of the recently-established Annual 
Review of Informablon Science and Ttechncl p fgy . 



2.2 MEASURES OF EFFECTIVENESS VERSUS MEASURES OF IMPACT 

A communication technique can be evaluated from two vleypoints: effective- 

ness and lip>act. Knowledge of effectlvenia;ss is a self-evident requirement; 

•• one most know how well the technique does the Job for >diich it was intended, 
and one must know what kinds of shortcomings may require attention. !Qie 
need, to appraise the Inpact of communication techniques and media may not be 
quite as self-evident, but it is equally or more important. Inpawt informa- 
tion tells wiiether and how new techniques and media affect the process of 
scientific and tech.nlcal communication. For example, one may introduce a 
bibliogrsphy into a field, as an information resource for its professionals. 
From an effectiveness standpoint, one mi^t look at the comprehensiveness 
or accuracy of the bibliography or at such factors as typographic clarity, 
etc. From an impact standpoint one would want to know what changes, if any, 
the availability of the bibliography made in the information-seeking or.’ 
information-sharing behavior of its users. Ihese changes will range from 
altered behavior towaird the library, colleagues, and other information 
sources previously relied upon, to the initiation of personal contacts with 
other professionals whose names come to the user's attention through the 
bibliography. Effectiveness measures are more narrow, well defined and 
short-term in nature. From a methodological standpoint, one does not 
ordinarily attempt to look past the immediate situation for subtle or long- 
range effects. On the other hand, jjgpact measures tend to be broader, more 

diffuse, and rather long-term in nature# Qhey often produce surprising 
Infofrmatlon of a so(rt that could not easily have been predicted. 

Nell -conceived impact studies can provide much more basic,, generallzable 
and useful kinds of ’knowledge of informat ion -seeking and information use 
behavior than effectiveness studies. This is especially true with respect 
to innovative communication techniques* With ’’settled" institutions or 
media, such as established journals, enough experience may have accrued so 
that the effectiveness measures cover the important questions and potential 
outcomes. However, when one is dealing with innovations, especially in 
technical areas where there is a hij^ rate of change, the experience base 
for accurately selecting the proper modes of effectiveness evaluation is 
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not yet developed. 2he discovery of such new modes is the most iiDportant 
element in sdspting present communicstion institutions Eucid forms to chsn^in^ 
times and in providing guidance for the development and financial support 
of new institutional forms. 



2.3 ASSESSING IHE IMPACT OF DIE ANNUAL REVIEW 

Annual reviews are the most ambitious and highly organized members of a 
class of comunication tools that atten 5 )t to serve as "condensed 
representations' of widespread activities. (Other members of this class 
are progress surveys, data con®ilations and state-of-the-art reports.) 
While no longer "innovative" in concept— there are over 500 annual, review 
series in existence --they are certainly evidencing an upsurge in attention 
and application. There is a widespread and growing conviction among 
professionals concerned with scientific and technical commonication that 
there should be much greater use of this kind of communication device . As 
one g*- .nernment official put it several years ago: 

Information science and technology have grown at such a pace in 
recent years that attempts to ^take stock^* of accomplishments seem 
even more important at times than new research and development.* 

More recently, strong support of critical, incisive progress reviews has 
been voiced by SATCOM, now completing an extensive study of ways in which 
the Federal government and the private sector can work together to improve 
scientific and technical communication. 



Interestingly enou^, the faith in progress reviews is based almost entirely 
considerations. A careful literature search has turned up only one 
enpirical study that involved the impact or effects of annual reviews. This 
study, part of the project on scientific communication performed by the 

^erican Psychological Association,** mentioned a number of values of the 
Annual Review of Psychology , but the project did not provide any information 
on changes in communication or professional effectiveness that accompany the 
introduction and continuing availability of an annual review. Because these 
changes are n,>cossarily dynamic, one could not expect to develop an accurate 
understanding of iiqpact behavior from a single study. Rather, one must 
expect to sainple from several— even many— stages, beginning as early in tho 
life of the innovation as possible. 



HELEN L. foreword. In: Cuadra, Carlos A., ed. Annual Review 

l^f Technology, vol. 1. Interscience, Hew York, 



**American Psychological Association. 
Exchange in Psychology. Reports of 
Washington, D. C. , 1965, Vol. 2, p. 



Project on Scientific Information 
work performed under NSF Grant -281. 
12-14, 245. 
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In 1966 an opportunity developed to saiiple for the first time the first 
stafexi in the use of a new annual review publication. In response to the 
need voiced by many professionals in the information science field for 
regular progress reviews, the National Science Foundation, the American 
Society for Information Science (then the American Documentation Institute) 
and System Development Corporation joined, in 1964, to establish the 

Annual Review of Information Science and Technology . The purj?ose of this 
series was and is to describe, relate, and critique the most liqportant work 
that has been reported in the field during each calendar year and to do this 
with sufficienb accuracy, coinprehensiveness, and objectivity that 
professionals in the field may rely on it as an authoritative and timely 
progress report. 

The inaugural volume of the Review was issued in September, 19^6. Judging 
from the favorable reviews in the journal literature and from the unexpect- 
edly hi^ sales, this volume passed the "effectiveness" test. However, no 
evaluation of the book was available from the standpoint of its "impact." 
Because the introduction of the Review provided a rather unique opportunity 
to obtain certain kinds of imp act -related data that couT.d very likely never 
be obtained at a later date, an Annual Review Impact Study was initiated. 



2,k ST?0DY GOALS AND LIMITATIONS 

As indicated earlier, SDC has been a co-sponsor of the Annual Review, and 
SDC personnel manage and edit the Review, on behalf of ASIS. Thus the 
goals of the study reflected interest both in the impant on professional 
communicators of this kind of publication and in the improvement of the 
Review per se . 

With regard to the first goal— assessing impact— it was clear that the goal 
could not be fully achieved, since no enpirical information on the 
professional communication practices of the survey sample was available 
prior to the survey, (ideally, impact studies should include both "before" 
and "af’ber" data.) Nevertheless, the survey could provide baseline informa- 
tion against which information obtained at a later point in time could be 
conpared. The information obtained in this study would, of course, be of 
interest in its own ri^t, quite apart from its value as a basis for future 
comparisons. 

The second goal— obtaining information for improving the Review— raised 
questions like these: 

1 ) How does reading the Review relate to other patterns of informa- 
tion use, and to personal attributes? 
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2) What specific functions does the Review serve and how is it 
likely to affect patterns of scientific cominunication (i.e., 
information-seeking and information use) among information 
scientists? 

3) What changes do readers feel would most iiqprove the usefulness 
and value of the Review? 

It was also hoped that the study might provide some preliminary guidance 
to funding agencies, professional societies, and other groups interested 
in supporting annual review publications* 

Bie approach and specific procedures used in addressing these goals are 
described in the following section. 



15 November 1968 



26 



TM-4125 



3. DESIGN AND EXECUTION OF THE SURVEY 



The main work in the Annual Review Impact Study was conducting a s\irvey of 
users and non-users of the Review. The survey itself involved three main 
tasks : 

(1) identifying a suitable sample of Review users and non-users; 

(2) determining, by means of questionnaires and personal contact, 
the attitudes sjid practices of these groups in. relation to 
information sources, including the Review, and 

(3) analyzing and interpreting the results. 

There are a number of pitfalls in all survey research, notably the danger 
that respondents will be affected by the knowledge that they are being 
measured. That is part of the reason why it has been said* that a survey 
is not first on a careful researcher's list of preferred information sources. 
Nevertheless, the disadvantages must be weighed against the advantages, such 
as low cost end ease of use. Researchers must resort to a survey when the 
needed information is not available by other means and when no natural 
event is going to make it available. Since the desired information about 
the use of the Review was not available from any source other than the first 
users of the book, the survey technique was chosen. Later, however, the 
impact of the review may become evident in references to it from significant 
new projects undertaken in the fields it covers. 

It was decided, at the outset of the study, that use of a mail questionnaire 
would be the most efficient and least expensive way to supply the bulk of the 
data. However, the study was planned so that the impact of the Review would 
be examined both by mail questionnaires and personal interviews. This 
enhanced the validity of the study design. 

3.1 SAMPLING 

To identify the users, business reply postcards (See Figure l) were sent in 
window envelopes to approximately 10,000 members of four professional societies: 
The American Documentation Institute (now the American Society for Information 
Science), the Special Libraries Association, the Association for Machine 
Translation and Computational Linguistics (now the Association for Computational 
Linquistics), and the Special Interest Group on Information Retrieval of the 



*Paisley, William J, "Some Decision Points in Survey Research" In: Studies in 
Journalism and Communications . Study No. 6; Decision Iteints in Mass Communi- 
cation Research— Survey. Content Analysis. Historical and Experimental Methods , 
edited by Donald L. Shaw, UNC, Chapel HilLl, N.C., June 19b?. 
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Association for Computing Machinery. These organizations were selected on 
the assumption that the largest concentration of Review readers would be 
found in them.* The postcard asked whether the person had seen, heard of, 
or used, the Annual Review, and which one of eight Job categories accounted 
for the major portion of his professional activities. The first question 
devided respondents into three groups: those person who had used the 

Annual Review, those who had only seen it or heard of it, and those who had 
not yet seen it or heard of it. (Throughout the remainder of this report, 
these groups will be referred to as Users, Seen/Head.s, and Not-Seen/Heards , 
respectively.) It was felt to be important to include non-users— the latter 
two groups— in the survey so that they could be compared with users. It was 
hoped that such a comparison, involving information-use practices and demo- 
graphic attributes, such as education and place of employment, would help to 
escplain why some professionals use the Annual Review and other do not. 

The Job categories on the postcard were intended to divide respondents into 
eight categories, each of which could be sampled to obtain the required num- 
ber of subjects for the questionnaires. However, almost 15 percent of the 

respondents checked more than one Job activity, so it was not possible to 
follow the original plan completely. Instead, questionnaires were sent to 
almost all Users. The Job activity data from the postcards were retained, 
for possible use later in the analysis. (Multiple responses were treated 
as though they comprised a separate new category. ) 

Almost 3700 postcards were returned, almost equally divided among the three 
groups. Figure 2 shows mailings and returns for the postcards. Because a 
final data set of about 2000 questionnaires was desirable, for purposes of 
analysis, and because the normally ejpected percentage of returned 
questionnaires for a professional sample is somewhat under 6o percent, an 
initial sample of about 3300 appeared necessary. With only a few exceptions, 
questionnaires were sent to all Annual Review Users. For the Seen/Heard 
and Not-Seen/Heard gro\ps, the two largest Job activity categories (library 
or information systems service, and administration of library or information 
systems, for each group) were randomly reduced to a more manageable size 
before the questionnaires were mailed. The total number actually mailed 
was almost 3000. 



•^Polling the members of four other professional associations was considered, 
but rejected, either because there would have been too much overlap with 
members of the four primary organizations (as determined by a quick telephone 
sample of Los Angeles members), or because the organization's activities were 
not closely related to those of ASIS. The four were: the American Associa- 

tion of Law Libraries, the Association of American Medical Libraries, the 
Division of Chemical Literature of the American Chemical Society, and the 
Society of Technical Writers and Publishers, 
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Dear colleague: 

With the assistance of the American Documentation Institute, Special Li- 
braries Association, and Stanford's Institute for Communication Research, 
System Development Corporation is trying to identify users of the Annual Re- 
viewof Informafion Science and Technology. The goal of the study is to deter- 
mine patterns of information medio use in different professions. 

We would appreciate your answering three brief questions and returning 
this postcard to us. Thank you. 

Linda Harris 
Study Director 



If your address is incorrect, please moke the appropriate change. 



Hove you seen or heard of the Annual Review? 

Yes No. 

Hove you used the Annual Review? 

Yes No 

Which one of the following best represents your ac- 
tivities during the post year? 

Library or information systems service 

Design/development of library or infor- 
mation systems 

Research in library or information sys- 
tems 

Research in language or language pro- 
cessing 

Administration of library or information 

systems 

Research management 

Taarhing OT educational research 

Marketing or sales 

Other 



Figure 1. Postcard Used to Identify Annual Review Readers 



3.2 DEVEIOPMENT AND USE OF THE QUESTIONNAIRE 



Separate questionnaires were developed for the User, Seen/Heard, and Ifot- 
Seen/Heard groLqps. The User questionnaire was the most detailed, containing 
25 questions. The questionnaire for the Seen/Heard and Nbt-Seen/Heard groups 
contained 15 and 12 questions, respectively. Ihe three questionnaires, shown 
in Appendix A, together with the corresponding cover letters, had 12 questions 
in common that were related to infoimatlon-seeklng and information-use 

behavior and to demograpMc attributes, so that the three groups 
could be compared on the same measures of reported behavior. The additional 
questions posed to the User group dealt with awareness, use, and impact of the 
Annual Review, 

In an attempt to obtain roughly comparable data for the Users and Seen/Heards, 
two Questions on the User questionnaire were rephrased to be suitable for the 
Seen/Heard questionnaire. 
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POSTCARDS MAILED 
Or ganization 

American Documentation Institute 
Special Libraries Association 

Special Interest Group on Infor- 
mation Retrieval of the Association 
for Computing Machinery 

POSTCARDS RETURNED 



Job Activity Category 
Library or information 
systems service 

Design/development of 
library or information 
systems 

Research in library of 
information systems 

Research in language or 
language processing 

Admini stration of library 
or information systems 

Research management 

Teaching or educational 
research 

Marketing or sales 

Students 

Other** 

Multiple Response 

Returned, with no response 
(=117 total) 

TOTAL 



1133 



Number 

2500, including 33 in Canada 

6500 , including 300 in Canada and 
59 in Europe 

1100 , including 37 in Canada 
500 , Including 8 In Canada 



Nbt-Seen/ 

Users Seen/Heard Heard 
334* 4l8* 



117 



1052 



1485 



TOTAL 

945 

398 



367c 



* Cut, randomly, to-150 before mailing questionnaires 

Ths larg 6 st group consisted of students ^ who were pieced in & seper&te 

category 



% 



Figure 2. Postcards Mailed and Returned 
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For exainple, on one question, Users saw this version i 

"Even If you have not read or skimmed all chapters of 
the Annual Review, which ones do you consider to be within 
your particular area of professional interest?" 

(the chapter titles were listed, verbatim) 

The Seen/Head group saw this version: 

The following is a list of Annual Review chapter topics* 

Please check the ones you consider to be within your 
particular area of professional interest." 

(the chapter titles were paraphrased, to 
indicate the chapter contents better) 

The prupose of these questions was to get information on the views and 
behavior of someone who was aware of the existence of the Annual Review, 
but who had not yet used it. 



Each version of the questionnaire was pre-tested and revised several times 
using ASIS and SLA members in the Los Angeles area. Most of the questions’ 
were pre-Soruct^ed (i,e,, provided a list of choices) to make the coding 
and data analysis easier; some open-ended questions • were included to stimulate 
suggestions for improving the Annual Review and to provide for responses that 
could not be anticipated when the questionnaire was written. 



The questionnaires were mailed from and returned to the Institute for Communi- 
cation Research at Stanford University, which was cooperating with SDC on the 
surv^. It was agreed that this use of the Institute address would increase 
the Ukelihood of receiving frank evaluations of the Annual Review. A summary 
report of the survey results was offered as an incentive for returning the 
questionnaire,* An original and one follow-\q> mailing were used, yielding an 
percentage return of 70fo (see Figure 3). The final data set contained 
?AT^I) questionnaires. Broken down by groups, the return rate was 6(yfo 

(N=64i) for Users, 70 ^ (N^512) for the Seen/Heard group, and 8 ?^ (N* 859 ) for 
the fet-Seen/Heards. The high response rate for the latter gro\q> might be 

attributed to the fact that their questionnaire was the shortest of the 
three versions . 



*Suraitiaries (see i^endix B) were requested by 8% of the respondent, Ihev 
were distributed in October, 1968 ." ^ 
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USERS 


SEEN/HEARD 


NOT-SSEN/HEARD 


FIRST MAILING 


Mailed out 


1031 


750 


1050 


Returned 


434 


339 


583 


SECOND MAILING 


Mailed out 


597 


4ll 


*167 


Returned 


207 


173 


276 


5 

TOTAL RETURNED 




64l ( 60 f>) 


512 (70^) 


859 ( 82 ^) 



Fig\ire 3. Questionnaires Mailed and Returned 



3.3 



GATHERING OF SUPPLEMENTARY DATA 



Data to supplement the questionnaires were obtained on three occasions The 
lirst was at the ASIS 30th Annual Convention in New York in October 1^7* 
by then, many questionnaires had been returned, and ASIS readers whose ’ 

intriguing were asked to atLnHSim^ group 
discussions to talk about the Annual Review and to offer suggestio^ f^ 

in^roving it in the l^ture. In addition, individual iSe^l^fw^re held 
^th readers. Because of the busy schedules of most convention attendees 
only about 20 interviews were held. Some of these sessions were tane ’ 
recorded, with the permission of the participants. ^ 

Supplementary data were also gathered, in Oetoher loAv ^4^ ^ ^ j t 

lishe^°^°+h^^A^^® headquarters of Encyclopaedia Britannica (the curren^pub^ 
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k, DATA ANALYSIS 

4.1 CODING AND .ANALYSIS OF F0RI4ATTABLE DATA 

The primary goal of the data analysis was to achieve an accurate and meaningful 
description of the 2012 replies to the questionaire. (it was decided to exclude 
from the analysis the information on job activites fi'om the business reply 
postcards. The questionnaires provided similar information, and since they 
were mailed after the postcards, the job activity information provided by the 
questionnaires could be presumed to be more current.) 

The analysis needed to take into account the fact that some of the questionnaire 
items involved pre-structured (formattable) data and others involved free 
response (nonformattable) data. The first data analysis procedure was to 
obtain an inventory of all responses to each question. On the basis of the 
inventory, some coirparison of the three respondet groups was made, and then 
decisions were made as to how to proceed with the rest of the analysis. 

Several inoportant conclusions are based on a comparison of demographic 
attributes and professional communication patterns for the three groups. 
Furthermore, some of the demographic attributes were used only for the 
intergroup comparison, and not in any other portion of the analysis, since 
they did not’ yield any information about communication patterns within each 
of the three groups. These attributes are membership in professional organi- 
zations, highest earned degree, year and field of that degree, and sex. 

The data inventory showed that no further analysis should be attenpted for 
the three questions shown in Figure 4. The replies to the first of these 
questions adequately indicated the types of information Users are obtaining 
from the Review, but since types of information are very general, further 
analysis of these figures would be neither fruitful nor justifiable. The 
response distribution for the second question showed that the "often-sometimes- 
never*' scale was unsatisfactory for making fine distinctions among Users, and 
that the list of activities was certainly not exhaustive. Thus no further 
analysis was attempted for this question, either. Respondents in all three 
groups were asked, in Question 25, for a description of their professional 
specialties. The replies varied so much in length and specificity that coding 
of this question was not attenqpted. More reliable information was obtained 
from replies to Question 24, regarding the respondents’ primary job activities. 
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12 . 



Have you obtained from the Annual Review itself information about 
any of the following? 



JLdean or theories 
juethods or procedures 
jre suits or data 
__an individual and his work 



_a system 
ja project 

_other (please i^cify) 



l4. How often have you used the information you've obtained from the 
Annual Review for each of the following activities? 



Often 



Sometimes 



Never 



25. 



Ordering documents or 
publications 

Searching the literature 
as a service to others 

Writing reports, articles 
and p^ers 

Preparing lectures or 
leeches 

Writing your own review 
of the literature 

Preparing a bibliogrephy 
or reading list 

Exchanging information 
with your colleagues 

Other (plea^se specify) 



Please describe the area of your professional specialty for the most 
time-consuming activity you listed in question 24 (e.g, library 

e^ineeri^^^etc information systems, language processing, 

f specialty for the second most time-consuming 

activity listed, describe it"too, consuming 

(This question not coded.) 



Figure 4, Questions not Subjected to Further Analysis 
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The second decision was to code l8 Seen/Heard respondents as Not-Seen/Heads 
because they replied that they had first heard of the Review as a consequence 
of receiving the questionnaire. 

To show major trends in the data more clearly, the categories for several 
variables were combined on the basis of a shared concept or theoretical 
orientation. For exaii 5 )le 3 in the question, "How did you first hear of the 
Annual Review?" responses in response categories 1, 2, and 3 in Figure 5, 
below, were combined, because they show that an ADI related channel was 
the first source of infoimtion for respondents marking those choices. 

Categories 7-12 were combined because, each of them alone did not contain 
enough cases to support a cross-tabulation analysis. They were included in 
the» analysis but not interpreted, because they are too heterogeneous to be 
inter'pretable Therefore the analysis was confined to patterns of 
communication within channels A-D. 

In some cases, a variable with several categories was simply dichotomized 
to form categories such as, " re sponse/non- response," or "high/ low." When 
this was necessary, the split was made closest to the median of the response 
distribution. 

Whenever the coding task was to record numerical answers, such as "number 
of chapters read in detail," a small sub-sample of questionnaires was used 
to determine the range of responses, and then categories were established 
so that the distribution of responses would approximate the normal curve 
as closely as possible. 

When the procedures were completed, the next major analysis technique used 
for the formattable survey data involved construction of frequency tables by 
comparing (cross-tabulating) criterion variables (such as number of Review 
chapters read) against classification variables^ (such as prlmaiy Job 
activity). Response percentages were computer for each cell of these 
frequency, or contingency, tables. 



^Defined by exhaustive and mutally exclusive attributes by which respondents 
are classified. 
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Figure 5. Response Categories for First Information 
Source about the Review 
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4.2 FORMATION OF INDEXES 



A broad questionnaire survey often requires more than one question to 
adequately probe an attitudinal or behavioral phenonenon. To measure 
such a phenomenon, the responses to a number of different questions are 
combined to form a single index, and the distribution of responses for 
the index is taken to represent the desired meas\ire. 

Three summative indexes were created from portions of the survey data. 
The first, which will be referred to the Professional Organization 
Indes, represents the total number of professional organizations in 
which each respondent is a member. This is generally considered to 
be a better indicator of professioned communication activity than an 
analysis of which respondents belong to each individual organization. 



The second summative index. Journals Read Regularly, is intended as one 
measure of extent of professional information use: it is a count of 

number of journals read regularly (almost every issue). The relevant 
question asked, ” Which journals do you read or scan regularly?” The 
rationale behind the question, however, was that a more accurate and less 
inflated report of journal reading would be obtained by requiring a 
specific list of titles. Nevertheless, coding this question proved 
to be extremely difficult because many respondents replied, ’’Too many 
to mention.” 

The third index reflects Diversity of Authorship. The basic question, 
posed to the three groups, asked how many different kinds of reports 
and publications a respondent had authored or co-authored within the 
past five years. Authorship of any of the six kinds of materials — books, 
chapters in books, journal articles, technical reports, unpublished 
papers, book reviews— was assigned a value of one, and the Index was 
created by summing the values for each kind of material. Thus, the 
maximum possible value on the Index for any one respondent was six. 



*In a summative index, all values of the variables to be combined are 
added and then the resulting distribution of values is divided into 
categories to form levels of the index. In a Boolean index, on the other 
hand, the levels are formed by combining values of the variables in 
all possible ways. 
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A fourth index, based on Boolean logic, measured total extent of readership 
of the Review. This index, the Read/Skim Index, was fomed as the sum of 
number of chapters read and proportion skimmed. The latter 
variable was defined in terms of the following ratio* chapters actually 
skinmed (Y)/chapters possible to skim after reading (X) chapters in detail. 
(See Figure 6. ) Skimming had to be defined this way because skimming and 
reading are mutually exclusive behaviors, and it is lnpossible to skim a 
chapter immediately after reaidlng it in detadl. Because the Read/ Skim Index 
is not a weighted combination of two vaurlables, a respondent who read one 
or two chapters in detail auid skimmed about nine was considered e<iulvalent, 
for analysis puiposes, to one who had read six to nine chapters and skimmed 
about five. 

4.3 POPING AMD ANALYSIS OF BONFORMATTABLE DATA 

The nonformattable data Includes replies to the three "open-ended" questions, 
(questivons 11, l6, and 17 on the User questlonnadre), replies listed in the 
"other" category for severad prestructured questions, and information 
obatlned from interviews with Annuad Review Users at three different meetings. 

The coding procedure for two of the three open-ended questions (U and 17) was 
tc .type, ’/erbsti!!*, ^ responses to e*ch question, and then, by looking at 
the verbata, to derive for each question a set of categories, based on some 
conceptuad scheme, that adequately represented the content of the replies. 
Question l6 was not coded. 

Figure 7 Illustrates the branching coding scheme used for question 11 
(Users), "For what purpose did you last use the Annual Review?" This 
question .Is dubbed the "critical incident" question, because it refers to 
the most recent Incidence (and therefore probably the most easily recalled 
incidence, as far as the reader is concerned) of Review use. 
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No, of Chapters 
Read in Detail 



0 = NR 



1-2 



3-5 



6-9 



10-12 



Response Percentage 



l6/o 

12^ 

10 ^ 

18^ 

hvjo 



No. of Chapters 
Skimmed 



and 



and 



and I 



and 



and 



r 0 =r NR ^ 



J 1-3 
< 4-6 
T-12 



Level of 
Read/Skim Index 



0 

1 

2 

4 

0 

1 

3 

5 

1 

2 

4 

5 

2 

3 

5 

5 



5 



Level of Read/Sklm Index 

0 

1 

2 

3 

4 

5 



Figure 6, Formation of the Read/Skim Index 
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^ 

BROAD CUT 

^ 1 

SYSTEMATIC BROWSING 

USE 



TASK-SPECIFIC TOPIC-SPECIFIC 



"For what purpose did you [last] use 
the Annual Review?" 




PROFESSIONAL USE 



STUDENT USE, 
CLASSWORK 



NARROW CUT 
>p ^ 



CHECKING A CHECKING A 

REFERENCE FACT 



st 

RESEARCH TASK 



J 

INFORMATION 
DISSEMINATION TASK 



INFORMATION 
SYSTEM TASK 



Figure 7. Coding Scheme for Critical Incident Question 
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A binary-decision scheme (two alternatives at a time) was developed to 
code the responses, some no more than two words long, some very 
detailed sentences. It was assumed that a reader who is putting the 
Review to a professional use can take a ’’broad cut" or a "narrow cut" 
through it. If he takes a narrow cut, he might be checking either a 
reference (citation) or a fact. This implies a very brief, purposive 
use of the Review. 

The broadest possible use of the book was defined to be browsing, and 
this includes responses such as "state-of-the-art," "current awareness," 
"general" or "background" information, and "review," The heterogeneity 
of the responses does not allow a further breakdown of this category 
into, for instance, a brushing -up or review function vs, a learning 
or orientation function. On the other hand, learning is implicit in 
many of the branches of this tree structure, so it would be difficult 
to attribute learning only to one goal of Annual Review use. 

Complementary to browsing is a systematic use of the Review that is 
still more general than the reference function. Any respondent who actually 
stated his purpose could fall into either the task-specific category 
(e.g., a project) or the topic -specific category (mention of a specific 
chapter title or topic within the scope of the book), (if a respondent 
combined both task- and topic-specific goals within the same response, 
the task-specific one was preserved, because these, in general, were 
more meaningful.) In turn, tasks were divided into research tasks 
(information-seeking in connection with explicit research), personal 
information dissemination tasks (e,g., use of the book in connection 
with the writing of a paper or proposal, with teaching or lecturing, 
or other inf ormation/communication activities), and information system 
tasks (e.g,, information flow activities, such as "selecting items for 
inter-library loan"). Five percent of the respondents indicated 
multiple purposes of their last use of the Annual Review, and if these 
purposes were not specific tasks, the first one mentioned was coded. 

Code categories were also developed for replies to Question 17 for Users, 
They are shown in Figure 8, 



* 

Funkhouser, G. Ray, "Binary Coding," Appendix VI, Final Report, 
Patterns of Adult Information Seeking , Sept., *66, Edwin B, Parker and 
William J, Paisley, Institute for Communication Research, Stanford 
University. (U,S,0,E. h 10 193; PN 2583) 
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17 . "In the future, what aspects of the Annual Review might 
be changed to make it more useful?" They are: 

Style of Writing 

1. Level of criticism 

2. Level oi detail 

Content of Volume 

3 . Range of coverage 

4. Addition of specific Chapters 

5 . Deletion of specific Chapters 

6. Summary Section 

Organization of Volume 

7 . Organization of Chapter Content 

8. Coordination among the Chapters 

Tools for Access 

9 . Sources of cited publications 

10. Prices of cited publications 

11. Indexes, appendices,, and bibliography 

Mechanics of Production 

12. Publication cycle 
13 • Typography 

Miscellaneous 

14. Price of the Annual Review 

15 . Other 



Figure 8. Code Categories for User Question I 7 
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A coding scheme was not developed for Question l6, shown in Figure 9» helow. 
Close inspection of replies to this question showed that it was misunderstood 
by many respondents. For exanple, replies that a chapter of the Review 
appearing monthly would be "more current" reflect a misinteipretation of 
the possibilities suggested: the chapter would still review the previous 

year's literature, but only in one topic area— and a different topic would 
be covered each month. Furthermore, both publication schedules were viewed 
as time-saving by respondents on both sides of the question. Because of the 
apparent ambiguities in this question, a Coding scheme was not devised for 
the replies. However, the most frequently cited reasons for preferring 
each publication schedule were detemined by inspecting the verbata. 



l6, "If one chapter of the Annual Review were to be published 
each month instead of collecting the chapters in book 
form annually, how do you think the value of the Annual 
Review would chang? A different chapter each month would 
be? (check one)" 



much less ^somewhat less ^equally ^somewhat more much more 

^useful useful useful useful useful 

Why? 



Figure 9, User Q^estion l6 

Many of the lists in the pre-structured questions on all three versions of 
the questionnaire "other" category to capture miscellaneous responses not 
anticipated when the questionnaire was written. Because the "other" 
responses, were so few (with one exception), they were not coded. 

The data gathered through individual and group interviews and discussions 
at the 1967 and 1968 ASIS conventions, and at the 1967 meeting at 
Encyclopaedia Britannica headquarters are essentially nonformattable and 
necessarily required qualitative, rather than quatitative, analysis. 
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5. RESULTS 

The survey provided & gre&b desQ. of d&b&y some of which defies e&sy interpre- 
tation or even easy description. The purpose of this section is to present 
the survey data and the results of analyses performed on it. For e?se of 
presentation, the results will be discussed in terms of eight major topics of 
interest in the study: 

• Characteristics of Annual Review Users 

• Acquisition of Annual Review Copies 

• Extent of Interest and Readership 

• Purposes Served by the Annual Review 

• Correlates of Previous Contact with Review- Cited Literature 

• Relationship of Readership to Other User Characteristics 

• Impact of the Annual Review 

• Suggested Areas for Improvement 

Some of the survey variables encompassed by particular topics above are inter- 
related. The final section of the report will attempt to synthesize and pro- 
vide a general interpretation of the findings. 



5.1 CHARACTERISTICS OF ANNUAL REVIEW USERS 



More than one third of the items on the questionnaires were addressed to the 
general question "Who are the users of the Annual Review and what are they 
like?" Sections 5»1»1 through 5»1»^ describe the findings on this question. 



5.1.1 Occunational Setting and Type of Work 

Figure 10 shows three questions dealing with respondents’ occupational setting 
and their type of work. These questions, like the others discussed in Sec- 
tion 5.1, were asked of all three groups, to permit comparisons between those 
who use the Annual Review and those who do not. (Both the Seen/Heard and the 
Not-Seen/Heard groups are considered to be non-users.) 
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Not- 

Seen/ Seen/ 
Users Heard Heard 



23 . In what type of organization are you currently en^loyed? 



university 

government agency 

industrial or business firm 

non-profit corporation, organization 

private foundation 

public library 

public school 

other, multiple response 

NR, not employed 

Totals 



2110 


20$ 


2% 


l4 


13 


11 


36 


42 


39 


13 


11 


9 


1 


2 


0 


1 


2 


3 


0 


1 


1 


6 


8 


10 


LOO^ 


1 

loojf 


2 

iSo^ 



24. Please rank the following job activities by placing a 1 in 
front pf your most time-consuming activity, a 2 in front of 
your second most time-consuming activity, etc. Ignore activ- 
ities you do not usually engage in. 

(The following are total response percentages, foi each cate- 
gory, regardless of the ranks assigned.) 



library reference, bibliographic service 
other library service (e.g. cataloging, 
acquisitions) 
research 

design or development 
management or administration 
programming 

consulting in your professional specialty 
teaching 

studying for a degree 
writing or editing 
publishing 

indexing or abstracting 

sales 

other 

Totals 



37^ 


50^ 


42^ 


26 


40 


36 


44 


39 


47 


44 


30 


32 


67 


65 


60 


l4 


15 


27 


37 


32 


36 


22 


17 


24 


13 


13 


l4 


36 


28 


29 


5 


4 


6 


20 


22 


l4 


4 


4 


4 


6 


6 


.. JL 



applicable) 



25 . Please describe the area of your professional specialty for 
the most time-consuming activity you listed in question 24 
(e.g., library science, library systems, information systems, 
language processing, engineering, etc.). 



If you have a different specialty for the second most time- 
consuming activity listed, describe it too. 



(This question was not coded. ) 



Figure 10. Questions Relating to Locale and Type of Work 
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The largest numbers of Annual Review Users are employed in Industrial or 
business firms; the next most common places of employment are universities, 
government agencies, and non-profit organizations » The occ\;patlonal settings 
for each of the two non-user groups follow the same pattern, but there is a 
greater preponderance of industried over university settings for the non- 
users than there is for the Users. 

There is somewhat more dissimilarity between Users and non-users in Job 
activities, as Indicated by the response percentages for Question 24. All 
three survey groups mention "management or administration" most frequently 
as an iinportant Job activity, but they differ noticeably on other Job activ- 
ities. Half of the Seen/Heard group, but only 37^ of the Users, report 
spending time in library reference work; there is a siinilar preponderance, 
for the Seen/Heard group, in the activity "other library service." In con- 
treist with non-users. Annual Review Users seem somewhat more likely to be 
engaged in writing or editing. 

The Not -Seen/Heard group is difficult to characterize. They are seemingly 
the most research-oriented and teaching-oriented of the three groiqps, but 
they are also much more actively engaged in programming than the User and 
Seen/Heard groups. Perhaps more people in the Not-Seen/Heard group tend to 
have rather specialized Interests that do not place them in contact with the 
information channels that brought the Annual Review to the attention of the 
User and Seen/Heard groups. Yet, the Not-Seen/Heards are very active in 
library work (more active than Users but less active than the Seen/Heard 
group), and it is surprising that, one year after the publication of the 
Annual Review, it had still not come to their attention. 

One can only speculate, at this point, on whether the two non-user groups will 
become Annual Review Users. Presumably, some of the Nbt-Seen/Heard group 
will become Users, since the survey itself had the effect of calling their 
attention to the Annual Review. Presumably, too, some of the Seen/Heard 
group will become Users, although the extent of change may not be large. 

The Seen/Heard group has a relatively greater library service orientation 
than the other groups, and it may include a large number of professionals 
who will continue to notice and catalog the Annual Review, without actually 
using it as a source of information. 



5.1.2 Education and Professional Background 

Three questions deedt primarily with educationsd and professional background. 
(See Figure 11.) The Annual Review Users differ from the other two groups 
in having relatively more M.L.S. degrees and relatively fewer B.A. or B.S. 
degrees as the highest earned degree. To put it differently, the User 
group has the largest number of professionally trained librarians (although 
Figure 10 above shows that more non-users than Users are actually engaged in 
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" i' 

fit 



m- 






\ Ur 








... 




Seen/ 


Not-Seen/ 


26 * 




Users 


Heard 


Heard 


What is your highest earn^ degree? 










^ d^ree 


1% 


0^ 


0? 




3#A# 1 


19 


27 


31 




B.L.S. . 


5 


6 


6 




MfA# ^ MfS# 


24 


22 


22 




M.L.S. 


29 


23 


15 




Ed»‘D* 


15 


12 


16 




other , 


5 


4 


2 




WR 


2 


6 


8 


' 


TotcLla 


lOO? 


loo?: 


100? 




Year earned: 


* 








before 1935 
1935-1946 


■ 3% 
6 


6? 

6 


4? ' 
6 




1940-1945 


7 


8 


6 




1945-195(5 


10 


7 


11 




1950-1955 


19 




15 


• 


1955 - 19 ^ 


15 


l4 


15 




1960-1965 


25 


24 


25 




after 1965 


11 


9 


10 




HR 

Totals 


4 

1005^ 


loo? 


8 

10^ 


27. 


Major field in which highest degree vcus earned: 








library science 


43 ^ 


38? 


28 ? 




psychology, sociology 


3 


3 


3 




linguistics, math, computer science 


7 


9 


17 




physics, chemistry 


13 


12 


7 




enginee;ring 


7 


7 


8 




biological sciences 


2 


3 


2 




education 


2 


3 


3 




business administration 


4 


3 


5 




English, history 


4 


4 


7 




other 


12 


11 


12 




m 

Totals 


loo^ 


IgI? 


8 

10^ 


22. 


How many years have you worked in the 


field of information 


1 science — 




i.e., documentation, library science. 


information 


systems research, . 




or related fields? 










less than five years 


24^ 


24? 


30 ? 




5-10 years 


29 


30 


26 




iO -15 years 


19 


19 


17 




15-20 years 


15 


8 


12 




20-30 years 


9 


13 


8 




over 30 years 


3 


4 


5 




NR 


1 


2 


2 




Totals 


Ioo5^ 


100? 


loo? 



ERJC Figure 11. Questions Relating to Education and Professional Background 
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library work). The User group also has more doctorates than the Seen/Heard 
group, though slightly fewer than the Wot-Seen/Heard group. The prepon- 
derance of non-library science B.A, and B.S, degrees in the latter group, 
together with the l6^ doctorates, suggests that this group is rather hetero- 
geneous and may include many newcomers to the information science field, e.g., 
computer specialists who are beginning to work in the area of library automa- 
tion. 

Some evidence for this speculation comes from the responses to Question 27. 

Only 28^ of the Not- Seen/Heard group had earned their highest degree in 
library science, compared with 38^ for the Seen/Heard group and 43^ for the 
Users. Correspondingly, twice as many of the Not-Seen/Heard group had earned 
degrees in linguistics, mathematics and computer science as had the respon- 
dents in the other two groups. These findings also support the notion that 
the Not-Seen/Heard group uses information channels different from those that 
brought the Annual Review to the attention of the Users, e.g., the computer 
literature, as opposed to the library or documentation systems literature. 

The responses to Question 22 provide additional evidence for the idea that 
the Not-Seen/Heard group includes more newcomers to library and information 
science than the other groups. The median experience level for Users is 
slightly higher than that for the Seen/Heard group, which experience level 
is, in turn, slightly higher than that for the Not-Seen/Heard group. The 
most interesting finding from Question 22 is that more than half of each 
group reports having worked in the field of information science for 10 years 
or less, in spite of the fact that the definition of information science used 
in the question was purposely made very broad. This finding suggests that 
the survey respondents perceive information science as a relatively new field 
of activity. The results could also mean that the respondents are relatively 
young, professionally. If "professional age" is taken from the year that the 
highest degree was earned, the data from Question 26 show that for all three 
groups, the median professional age is about 10 years. Indeed, this is a 
young sample. 



5.1.3 Professional Activity 

Four questions sought information regarding the respondent's level of pro- 
fessional activity, i.e., membership in professional societies and use of 
professional communication channels. They are shown, together with response 
percentages, in Figure 12. 

Users show the highest levels of professional activity. Seventy- nine percent 
of the Users belong to ASIS (formerly ADl) — a result expected because ASIS 
co-sponsored the Annual Review, and acquisition of the Review was made easier 
for ASIS members. Only 45^ of the Seen/Heard and 19^ of the Not-Seen/Heard 
groups belong to ASIS. Presumably, if more persons in these two groups had 
the interests and motivation to belong to ASIS, they would also be Users. 
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? 






Seen/ 


Not-Seen/ 






Users 


Heard 


Heard 


18 . 


To which of these professional organizations or 


associations in the 




field of information science 


1 do you belong? 






ADI (now ASIS) 


79f> 


45^ 


1^ 




SLA 


50 


56 


47 




ALA 


24 


22 


16 




ACM 


22 


20 


34 




AMTCL 


5 


6 


12 




Totals 


(Not Applicable) 




TOTAL NUMBER of Organizations, including others 


not specifically 




coded due to extremely low response percentages 


• 

• 




PROFESSIONAL ORGANIZATEONAL INDEX: 








none, NR 


1 % 


QPjo 


1^ 




one 


20 


33 


44 




two 


31 


36 


33 




three 


27 


22 


13 




four 


13 


5 


5 




five 


6 


2 


2 




six 


1 


1 


1 




seven or more 


1 


1 


1 




Totals 


100^ 


loo^ 


100% 


21. 


How many local or national r 


irofessional meetings (conventions. 




conferences, synqposia, etc.) 


have you 


attended during the past year? 




none, nr 


% 


1 % 


16 % 




one-two 


33 


4o 


43 • 




three-five 


39 


29 


26 




six or more 


-12- 


16 






Totals 




100% 


100% 




At how many of these meetings did you 


present a paper? 




none, NR 










any response 

Totals 


A 







Figure 12. Q^estions Regarding Professional Activity 
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Users 



Seen/ Not-Seen/ 
Heard .. Heard 



19. Wliich journos and other scientific periodicals do you read or scan 

bef^*oy« every issue)? (Specific Journals num- 

oe^^r 50 different ones, and were not coded. HUMBEB of 

"JOURNALS READ REGULARLY" INDEX: 

none, NR 
one- four 
five- seven 
eight-eleven 
twelve or more 



20 , 





13? 


10? 


31 


33 


42 


28 


23 


21 


12 


17 


13 


21 


Ik 


l4 


100? 


Too? 


15^ 



Totals 



i^roximtely how many of each of the foUowing materials have vou 
authored or co-authored in the past five years? ^ 

(The nu mber of materials authored was not counted: all resnonses 
were taken to be of equal value.) responses 

books 

chapters in books 
journal (or periodical) 
articles 

technical reports 
unpublished papers 
book reviews 

Totals 




32 
3 ^ 

15 

(Not Applicable) 



DIVERSITY OF AUTHORSHIP INDEX: SUMMATION OF ABOVE RESPONSES: 

none, NR 
one 
two 
three 
four 
five 
six 



Totals 



23? 


4o^^ 


39? 


21 


23 


23 


20 


17 


19 


17 


11 


10 


12 


5 


5 


5 


3 


3 


2 


1 


1 


Too? 




T^ 



Figure 12. Questions Regarding Professional Activity 

(Continuation) 



ERIC 



15 November 1968 



50 



TM-4125 



The Not-Seen/Heard group has a great many more members in ACM and AMTCL than 
the other two groups, a finding consistent with the earlier comment that the 
Wot-Seen/Heard group is more computer- oriented than the other groups. Inter- 
estingly enough, SLA has the most members of any professional organization 
among both the Seen/Heard and Not -Seen/ Heard groups, and half of the Users 
belong to it, too. Among the SLA respondents, the percentages of Users, 
Seen/Heeu:d, and Not -Seen/ Heard were much more evenly balanced than among the 
ASIS members. This suggests both that the interest patterns of the SLA mem- 
bership are more heterogeneous than those of the ASIS membership and that 
these interests are not as closely mapped by the Annual Review as are those 
of ASIS members. 

Analysis of the data shown in Figure 12 showed that, on the average, persons 
in the User and Seen/Heard groups belong to one more professional organiza- 
tion than those in the Nbt-Seen/Heard group. Users, on the average, attended 
more professional meetings during the last year and presented more papers 
at these meetings. The average number of journals read regularly was higher 
for Users, as were the different kinds of materials authored during the past 
five years. 

This general inpression of greater professional activity by Users is brought 
out more clearly in Figure 13, which compares the three groups on four pro- 
fessional activities. The statistically significant result is that the User 
group — and within that group, those high on the Read/Skim Index*— demonstrates 
the highest activity on each of the four measures. 

From Figure 13, it is evident that the "low" and "high" Users* ^ differ most 
on number of journals read regularly and number of professional meetings 
attended. Yet the three major groups differ from each other most on the 
index of Authorship Diversity, and on presentation of papers at professional 
meetings. An explanation for this may lie in the fact that journal reader- 
ship and meeting attendance tend to involve exposing oneself to information, 
that is, information input , whereas the other two variables reflect informa- 
tion production, or output of information. It is probably easier to absorb 
than to produce information, and maybe that is why information outputs 
better discriminate among the three groups. However, among the Users alone, 
information inputs are the better discriminators, probably because amount of 
information absorbed is a function of one's total network of conqplementary 
information sources. 



The Read/Skim Index was defined in Section 4. 

Based on dichotomizing the Read/Skim Index with 59?^ of the respondents in 
the "low" category, and 4l% in the "high" category. 
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USER 

Read/Skim Index 
’ High Low 


SEEN/ 

HEARD 


JOURNALS READ REGULARLY 








Five or More 
Less than Five 

TOTALS 


68^ 

100^ 


Ul* 

looi^ 


3 H 

46 

100^ 


DIVIi^ITY OF AUTHORSHIP 






■ 


Two or More Types 
Less than Two 

TOTALS 


6o^ 

4 o 

100^ 


5356 


M 

6s 

loo? 



NUMBER OF PROFESSIONAL MEETINGS 
ATTENDED DURING PAST YEAR 



Three or More 


6 ki 


53 ? 


46 ? 


Less than Three 


36 


4 ? 


3 k 


TOTALS 

PRESENTATION OF PAPERS AT 
MEETINGS 


Too? 


100? 


Too? 


One or More 


4 i? 


M 


19 fo 


None 




66 


81 


TOTALS 


100? 


loo? 


Too? 


N= 


261 


380 


512 



( = 641 ) 



NOT-SEEN/ 

HEARD 



48 ^ 

100 ^ 



39ii 

6l 

lOOjt 



To^. 



19 ^ 

81 

100 ^ 



859 



Figure 13 . Professional Activities of the Three Groups 
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The data in Figure 13 indicate no tendency for the Annual Review to serve as 
a substitute for journals; Users who read and skim it extensively also read 
journals extensively. As has been observed in previous studies of scientific 
communication, reading correlates with reading, and the high correlation 
observed here would tend .to mask any tendency for some individual readers to 
read the Annual Review instead of journals. 

To provide information on the use of professional communication channels. 
Question 15 asked respondents to assess the importance to them of a number of 
information sources. Figure ik shows the results fur all three groups. The 
list of information sources in the Figure includes "formal" information 
channels (e.g., technical reports, journal literature), informal channels 
(e.g., colleagues, professional meetings), and channels that often require a 
combination of formal and informal techniques for effective communication — 
e.g., preprints. 

The intent of Question 15 was to have respondents rate the information 
sources independently of each other, not in order of importance. Thus, we 
cannot compare the percentage figures and conclude that Users value col- 
leagues within their own place of employment more than they do outside col- 
leagues. The correct interpretation would be that 44^ of the Users value in- 
house colleagues as a very important information source, and a smaller percentage 
(40%) of the Users value outside colleagues as a very important informiation 
source. (One must also remember, of course, that we are discussing the value 
of these information sources in relation to current work. ) 

The three groups of respondents can justifiably be compared for each individ- 
ual information source. For example, in-house colleagues are "very inpor- 
tant" to more persons in the Not-Seen/Heard group than in either of the other 
two groups. Since the Not-Seen/Heard group is generally less active than the 
other two groups in their information use, it is understandable that they 
would rely more heavily on nearby information sources, easy to use. The 
situation is difficult to interpret for outside colleagues, because of oppo- 
site trends in the "very important" and "moderately important" colvimns. Here 
we find that more Users than Seen/Heards and Not-Seen/Heards consider outside 
colleagues as "very important" information sources. 

In comparison with the other groups, Users also tend to place greater value 
on professional meetings, technical reports, and preprints (together with 
outside colleagues) as very important sources of information. The Not-Seen/ 
Heard group tends to value books and reprints (together with in-house col- 
leagues) as very inportant information sources. The Seen/Heard group appears 
to be less active than the User group in their use of various information 
sources. Like the Not-Seen/Heard group, they place somewhat more value on 
books than Annual Review Users do. 

In general, the differences among the three groups shown in Figure l4 reflect 
some of the same differences in information-use behavior that were suggested 
by other data from the survey. Annual Review Users are most apt to 
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15 • How important is each of the following information sources in helping to provide 



Colleagues within your 
own place of employment 



Colleagues outside your 
own place of employment 



Professional meetings 
and synrposia 



Journal and periodical 
literature 



Otechnical reports 



Books, Qfexthooks 



Bie Annual Review 

Preprints of periodical 
ai*ticles 



Reprints of periodical 
articles 



Abstracting/ indexing 



Bibliographies 



"Other” 



ERIC 





Very 

Inportant 


Moderately 

Inportant 


Of Little 
or No 

Inportance 


Do Not Use 
!Ihis 

Information 
Source (4NR) 


U: 






12^ 


8^ 


s/H: 


• 4l 


31 


IT 


11 


N-S/H: 


52 


32 


11 


5 


U: 


ko 


44 


T 


9 


S/H: 


28 


48 


l4 


10 


N-S/H: 


26 


51 


IT 


6 


U: 


31 


52 


11 


6 


S/H: 


27 


50 


IT 


6 


N-S/H; 


18 


51 


25 


6 


U: 


63 


32 


2 


3 


S/H: 


6l 


33 


4 


2 


N-S/H; 


58 


34 


6 


2 


U: 


43 


39 


8 


10 


S/H: 


28 


35 


19 


18 


N-S/H: 


30 


39 


18 


13 


U: 


18 


55 


20 


T 


S/H: 


21 


45 


25 


9 


N-S/H: 


36 


4o 


19 


5 


U: 


24 


58 


10 


8 


U: 


13 


24 


26 


37 


S/H: 


10 


21 


24 


45 


N-S/H: 


8 


21 


28 


43 


U: 


12 


34 


27 


27 


S/H: 

1 


11 


32 


28 


29 


N-S/H:. 


IT 


31 


28 


24 


U: 


28 


kQ 


IT 


15 


s/H: 


29 


28 


21 


22 


N-S/H; 


30 


25 


20 


25 


U: 


22 


25 


20 


13 


S/H; 


23 


36 


22 


19 


N-S/H; 


26 


34 


25 


15 


U; 


2 


1 


0 


97 


S/H: 


2 


1 


0 


97 


H-s/H: 


8 


2 


1 


89 



Figure l4. Importance of Various Information Sources 



15 November 1968 



54 



TM-4125 



consider informal channels as inqportant information sources— and the corrobo- 
rating data (extensive authorship, and meeting attendance) indicate that 
these channels are indeed used more by users of the Annual Review than by 
non-users , 



5.1.4 Incidental Characteristics of the Sample 

For such a large-scale survey, and such a long questionnaire for Users, the 
overall response rate of 67 ^, as shown in Figure 15 , is excellent. Response 
rates between 50 fo and 60 ^ are normally expected for a san 5 >le of professionals 
like this one. The percentages of returned questionnaires appear to be 
inversely related to the length of the questionnaires. More persons in the 
Not-Seen/Heard group returned questionnaires than did those in the other two 
groups, probably because the Not-Seen/Heard questionnaire was the shortest. 

About 60^ of the respondents were males, this finding perhaps reflecting the 
membership characteristics of the organizations polled. The highest propor- 
tion of males was found among the Users, and the lowest in the Seen/Heard 
group. One explanation for this might be that the highest percentage of 
practicing librarians is in the Seen/Heard group, and the majority of prac- 
ticing librarians are women. 



QUESTIONNAIRE RESPONSE RATE: 


User 


Seen/ 

Heard 


Not-Seen/ 

Heard 


Received after first mailing 


M 




525S 


Additional questionnaires received 


18 


23 


25 


after second mailing 




■ 




Total Response Rate (6?^) 




m 


1% 


Initial N = 


1133 


752 


1113 


28 . Your Sex : 


Male 


oo 


’M 


61% 


Female 


32 


1*4 


39 




Figure 15. Incidental Characteristics of the Sample 
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5.2 ACQUISITION OF ANNUAL REVIEW COPIES 

The first question inquired into the ways in which respondents in the User 
group acquired copies of the Annual Review. (See Figure l6.) Responses to 
the question show that almost 6o^ of the readers used a copy that was more 
or less permanently available to them after they had put forth the initial 
effort to obtain it. Twenty- five percent of the Users borrowed the Annual 
Review through a library. This suggests that it is, in^wrtant to have the 
Annual Review easily available, for more effective use. 

Another question (No. 2 on the User questionnaire) asked both Users and the 
Seen/Heard group how they first heard of the Annual Review, me responses 
point up some interesting differences between Users and Seen/Heard respon- 
dents. About half of the Users first heard of the Annual Review through an 
ADI (ASIS) channel— announcements sent through the mail or made at the 1966 
ADI Annual Meeting, where Volume I was displayed and sold. Once information 
about the Annual Review was obtained from ASIS, some people were apparently 
interested enough to obtain the book. Almost 80^ of these people were ASIS 
members. In contrast, only l4^ of the Seen/Heard respondents first, heard 
about the volume through ASIS information channels. They were much more 
likely to have heard of the book through a colleague, or a publisher’s 
announcement or advertisement. One plausible hypothesis for the difference 

is that the Seen/Heard group is not greatly interested in information science. 
This group would, of course, contain relatively few ASIS members and would 
therefore hear about the Annual Review primarily through publisher announce- 
ments and advertisements in a variety of trade publications. The hypothesis 
receives support from the finding, reported earlier, that the Seen/Heard 
group includes more professional librarians and library service-oriented 
personnel than the other groups. 

The data in Figure I7 , which shows sources of Annual Review copies for people 
in different occupational settings, suggest that university personnel are likely 
to expand their personal libraries by purchasing the Annual Review, while 
employees of government and industry tend to rely on access to company -pur- 
chased copies. This is also shown by the fact that the company library is 
tha main source for the 31^ who borrowed a copy and who are employed in in- 
dustry. Approximately the same proportion of people in a university, in 
government, and in industry borrowed the Annual Review copy they used. 
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Users 

1. How did you obtain the Annual Review copy you 
have used? 



purchased a personal copy 31^ 

used a desk or office reference copy purchased 
with company or staff funds 28 

borrowed it from an individual 6 

borrowed it from a university library 10 

borrowed it from a company library 15 

other 9 

no response, don’t know, can’t remember ^ 

total lOOfo 

* 

2. How did you first hear of the Annual Review? 

preliminary announcement of contract award 11^ 

announcement prior to I966 ADI Meeting 21 

1966 ADI Meeting IT 

journal advertisement 7 

publisher’s advertisement 7 

colleague 12 

book review 2 

citation in a bibliogr^hy 1 

library ^ 

bookstore 0 

other, multiple response " 8 

no response, can’t remember 10 

totals lOOfo 



Seen/Heard 



6 

4 

8 

16 

8 

3 



2 




Question number 1 on the 



Seen/Heard questionnaire. 



Figure 16. Questions Concerned with Noticing 
and Obtaining the Annual Review 
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EMPLOYER 


Purchased 
a Personal 
Copy 


Used an 
Office 
Reference 
Copy 


Borrowed* 
a Copy 


Other 


Tot'als 




UNIV. 


36^ 




32^ 


V^lo 


100^ 


175 


OOV’T. 


29 


32 


30 


9 


100 


92 


INDUSTRY 


24 


34 


31 


11 


100 


229 


OTHER»^* 


34 


29 


29 


8 


100 










TOTAL NUMBER 


OF USERS 


, 641 ; 



^Borrowed from an individual, a university library, or a company 
library 



E.g., non-profit corporation. 



Figure 1?. Sources of Annual Review Copies 
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5.3 EXTENT OF INTEREST AND READERSHIP 

Before Volume I of the Annual Review was published, there was considerable 
speculation by the editorial staff regarding the likely range of interests 
of the audience and the extent to which a given user would read the Annual 
Review. The hope, of course, was that not one but several chapters would be 
of interest. Two questions, shown in Figures 18 and I9, elicited information 
on this topic, Question 3 in Figure 19 was intended to yield information 
that would determine the match between the Annual Review's coverage of topics 
in information science and respondents' interests. (The chapter titles were 
listed for Users and paraphrased for the Seen/Heard group. ) Users cited an 
average of five chapters they considered to be within their fields of inter- 
est, while Seen/Heard respondents cited an average of four. 

The Users' interest patterns appear to be different from those of the Seen/ 
Heard group. Users named the chapters on Library Automation and on Informa- 
tion Needs and Uses as the two most closely related to thei^.* interests, while 
the Seen/Heard group placed these third and ninth, respectively. The Seen/ 
Heard respondents considered the chapters on File Organization and on Infor- 
mation Centers and Services the top two in their fields of interest. Both 
groups coLSidered the chapter on Automated Language Processing the least 
germane to their fields of interest. Some caution is needed in interpreting 
the rankings shown in Figure 19» In some instances (for example, in the area 
of information needs and uses) the paraphrasing may have been less than ideal 
and may have influenced the judgments in unpredictable ways. 



Question h (Figure 18) examined the extent to which those in the User group 
actually read the Annual Review. The responses indicate that Users read a 
good portion of the volume. The median number of Annual Review chapters 
read in detail was approximately four, while the median number skimmed was 
also approximately four. Thus, the approximate number of chapters read and 
skimmed by a typical User was eight. This might be an underestimate, 
because some readers may not have had Volume 1 very long. 



4. Of the above 12 chapters, how many have you: 



Read in detail? 




Skimmed? 




none, NR 


19fo 


none, NR 


12% 


one -two 


23 


one -three 


25 


three-five 


34 


four- six 


35 


six- nine 


17 


seven-twelve 


28 


ten-twelve 

Total 


lOOPjo 


Total 


100^ 



Figure 18. Question Concerning Extent of Annual Review Readership 
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3. Even if you have not read or skimmed all chapters of the Annual Review, which ones do 
you consider to be within your particular area of professional interest? 


• 


Users 
^ Rank 


Seen/Heard 
- ^0 Rank 


Professional Aspects of Information Science and Technology 
(Educational Programs and Trends in Information Science)* 


h7i 


7 


29fo 


9.5 


Information Needs and Uses in Science and Technology 
(Behavioral Studies of Information Needs and Uses) 


62 


2 


29 


9,5 


Content Analysis, Specification and Control 
(Analysis, Description and Indexing of Document Content) 


39 


9 


51 


4- 


File Organization and Search Techniques 


58 


4.5 


6o 


1 


Automated Language Processing 

(Automated Language Processing and Computational Linguistics) 


25 


12 


20 


12 


Evaluation of Indexing Systems 
(Evaluation Studies of Indexing Systems and Terminology) 


59 


3 


47 


5 


New Hardware Developments 

(New Hardware Developments for Information Systems) 


33 


11 


34 


8 


Man-Machine Communication 

(Man-Machine Communication and Interaction Studies) 


38 


10 


28 


11 


Information System Applications 


57 


6 


42 


6 


Library Automation 


64 


1 


53 


3 


Information Centers and Services 


58 


^.5 


58- 


2 


National Information Issues and Trends 


45 


8 


36 


7 



Some of the chapter titles were paraphrased for the Seen/Heard questionnaire, 
to aid respondents who might not be familiar with the Annual Review contents. 



Figure 19* Question on Range of Interests in Annual Review Content 
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The relationship between reading and skimming is shown in Figure 20. The 
figure is read as follows: Of those persons who read 2 or fewer chapters 

in detail, 43% skimmed three or fewer chapters, 284 skimmed four to six chap- 
ters and 294 skimmed seven or more chapters. The 64 figure on the lower 
right hand cell suggests that some respondents, contrary to the questionnaire 
instructions, included the introductory chapter in their counts. Only by 
doing this could they have read six chapters in detail wid skimmed seven more. 

The most notable inference to be drawn from the figure is that medium reading 
is strongly correlated with high skimming; Users who seek specific informa- 
tion in the Annual Review are likely to skim much of the rest of the book. 



Number of Chapters 
Read in Detail 


Number of 


Chapters Skimmed 


Total 


N = 


3 or fewer 


4-6 


7 or more 




2 or fewer 








1004 


269 


3 - 5 


2ki 




\2$ 


ioo4 


218 


6 or more 


hii 




% 


ioo4 


134 










Total No. 
of Users =* 


641 



Figure 20. Relationship Between Reading and Skimming 



An alternate indicator of exposure to the book is the Read/Skim Index, Fig- 
ure 21. The skewed distribution of responses for the Read/Skim Index (4l4 
are in the highest category, which represents maximum reading and skimming) 
might suggest a possible misinterpretation of the question. Some readers may 
have mistakenly thought that the number of chapters read and the number 
skimmed should add to the total number of chapters in the Annual Review, and 
,this would have caused them to be placed in the highest category on the Index. 
However, it is also plausible that Users who are motivated to be fully ac- 
quainted with the material in the Annual Review might tend to skim all chap- 
ters that they do not read in detail; thus, they might legitimately have read 
and skimmed a total of 12 chapters. 

Additional comments on extent of readership are made in Section 5.6, which 
discusses the relationship between Annual Review readership and several other 
User characteristics. 
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Reading chapters chapters chapters chapters chapters Reading 



35 ^ 





Figure 21. Distribution of Responses for Reading, 
Skimming, and the Read/Skim Index 
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5.4 PURPOSES SERVED BY THE ANNUAL REVIEW 



To find out how well the Annual Review serves reference, current awareness, and 
other important information functions, both a structured question and an 
open-ended "critical-incident** question were included on the Users’ ques- 
tionnaire. The critical- incident question (**For what purpose did you [last] 
use the Annual Review?**) was asked first, to prevent response biases from the 
structured question. Figure 22 illustrates the coding scheme used for the 
critical-incident question, and response percentages for each branch. The 
responses indicate that the major professional uses of the book are for ref- 
erence, browsing, information- seeking on a specific topic, and information- 
dissemination tasks, such as teaching and writing. 

Replies to the structured question (**In general, how useful has the Annual 
Review been to you in serving each of the following [nine! goals?**) followed 
similar patterns, as indicated in Figure 23. **Keeping u^— both in the 
respondents' own areas and in peripheral areas--was the use for which Users 
found the Annual Review most valuable. Another Important use appeeirs to be 
"learning," which is comparable to systematic, **topic-specific** information- 
seeking in the critical-incident question. Learning is implicit in other 
categories of the critical-incident question, too, such as ** student use,** 
"research task,** and **browsing.** In fact., the **learning** and **keeping up** 
motives suggest one basis for the strength of the browsing response (34^) 
observed for the critical-incident question. The Annual Review probably 
plays a major role in the continuing education of its readers— a role that 
was intended in the original planning of the book. 

The use of the Annual Review for specific reference is not as prominent as 
its use for current awareness and learning, but it should be remembered that 
the survey dealt only with Volume 1. At this writing. Volumes 2 and 3 have 
been published; in all probability, use of the Annual Review volumes for 
specific reference has grown with the publication of the additional volumes. 

Almost 40^ of the Users find the Annual Review useful for **reading the orig- 
inal literature more selectively"— a result that clearly indicates the impact 
of the Annual Review. Other evidence of impact comes from the high percent- 
ages of Users who indicated that, after reading the Annual Review, they tried 
to obtain some of the publications cited and that they re-examined publica- 
tions they had already read. (Section 5*7 provides a more detailed discus- 
sion of impact.) 

Although replies to these two questions are quite similar, the response per- 
centages cannot be strictly compared because of the differences in the word- 
ing of the questions: a reader may not have found his last use of the book 

to be particularly valuable . Readers undoubtedly use the Annual Review for 
additional purposes, which they may not have remembered as they answered the 
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"For what purpose did you [last] use 
the Annual Revilew?" 




PROFESSIONAL USE 

( 92 !^) 



i 

BROAD CUT ( 7856 ) 



i 

SYSTaiATIC 

USE 



1 

BROWSIHG 

(3W 



i 1 

TASK-SPECIFIC TOPIC-SPECIFIC (l8^) 

( 26^)1 

r r 



RESEARCH TASK (T^) 



STUUENT USE, 
CUSSWORK (5^) 



1 



NARROW CUT (iW 



i 

CHECKING A. 
REFERENCE (8^) 



1 

CHECKING A 
FACT (6^) 



1 



INFORMATION INFORMATION 

DISSIMINATION TASK (l8^^) SYSTEM TASK (l^) 



Figure 22. Coding Scheme and Response Percentages for 
Critical Incident Question (N =* 55 I cases) 
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13. In general, how useful has the Annual Review been to you in serving each 
of the following goals? 

Have Not 
Actually 
Of Used It 
Moder- Little for This 





Very ately 
Useful Useful 


or 

No Use 


Function 
Yet (+NR) 


Totals 


Keeping up with current work in 
your own areas of interest 


33i 


hJi 


% 




loafo 


Keeping up with current work in 
peripheral areas of interest 


35 


39 


6 


20 * 


100 


Checking on particular projects 
or ideas 


19 


36 


14 


-2L. - 


100 


Checking on activities of 
individuals 


5 


17 


16 


AL. = 


100 


Learning about an area not 
within your professional specialty 


22 




8 

V 


4l * 


100 


1 i 

Readily the original literature 
more selectively 


-iL. 


24 


• 15 


46 * 


100 


Identifying areas of information 
science that require further 
research 


14 


20 


10 


* 


100 


Allocating research and develop- 
ment funds more effectively 


2 


4 


10 


84 


100 


Other (please specify) 


4 


2 


0 


94 » 


100 



N =: 64l 




Figure 23. Purposes Served by the Annual Review 
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questionnaire. Furthermore, many respondents probably had not found suffi- 
cient time to use the book in many different ways by the time they replied to 
the questionnaire. 

Figure 24 elaborates the goals listed in Question 13 in terms of the Read/Skim 
Index. One would read the figure this way: 32^ of the 129 readers who found 

the Annual Review of little use for keeping up in their own areas of interest 
were high on the Read/Skim Index--i.e., they read and skimmed the book exten- 
sively. The rest of those 129 readers (68^) were low on the Read/Skim Index. 
The major conclusion to be drawn from Figure 24 is that the higher the useful- 
ness indicated for each of the purposes, the higher the readership— a conclu- 
sion that was to be expected. The relationship between readership and useful- 
ness is statistically significant for all eight uses; however, the results of 
the statistical tests do suggest that reading and skimming are "functional 
equivalents" for some uses of the Annual Review. In order to "keep up," for 
example, a User may skim some chapters of the Annued Review, and read others. 



The eight uses of the Annual Review were cross-tabulated against the informa- 
tion sources listed in Question 15 (See Section 5. 1.3)— that is, colleagues, 
professional meetings. Journal literature, books, etc. The goal of this 
analysis was to determine how the Annual Review is used by people whose 
typical information patterns are print-oriented, colleague-oriented, etc. 

For exanple, how would someone who relies on colleagues and professional 
meetings as bis main information sources use the Annual Review? Unfortu- 
nately, no significant differences in use patterns were evident regardless 
of typical use of any of the ten information sources. Since other studies of 
information channel use indicate that use of an additional information tool 
such as the Annual Review should make a difference in the way other information 
sources are perceived and used, it seems possible that the questions asked were 
not sufficiently sensitive to detect small differences. 

One of the questions posed to respondents in the Seen/Heard group was designed 
to parallel Question 13 . As shown in Figure 25, these respondents would also 
use the Annual Review predominantly for current awareness, rather than for 
reference. 

The analysis of two additional questions for Users, shown in Figure 6, was 
limited to an inspection of the percentage results— i.e. , no cross -tabulations 
were performed. The results show the types of information obtained from the 
Annual Review, and activities associated with use of the Review. Users most 
often found information on ideas or theories and methods or procedures. The 
fact that only 24^ of the Users found information on results or data may ex- 
plain why. a major suggestion for improvement of the Review (discussed in Sec- 
tion 5.8) was to provide more information on results of the studies that are 
cited. 
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LEVELS OF USEFULNESS 
OF THE ANNUAL REVIEW FOR: 


PERCENTAGE 
HIGH ON READ/ 
SKIM INDEX 


PERCENTAGE 
LOW ON READ/ 
SKIM INDEX 




KEEnNG UP IN OWN AREA 
Low 


32 % 


68^^ 


129 


Medium 


39 


6l 


298 


High 


48 


52 


214 


KEEHNG UP IN OTHER AREAS 


Low 


24 


76 


166 


Medium 


46 


54 


248 


High 


48 


52 


227 


CHECKING ON PROJECTS 


Low 


38 


62 


287 


Medium 


37 


63 


233 


High 


55 


45 


121 


LEARNING 


Low 


34 


66 


313 


Medium 


42 


58 


186 


High 


54 


46 


1U2 


IDENTIFnNG RESEARCH GAPS 


Low 


37 


63 


427 


Medium 


47 


53 


126 


High 


50 


50 


88 


CHECKING ON INDIVIDUALS 


Low 


39 


6l 


499 


Medium 


44 


56 


108 


High 


56 


44 


34 


READING THE ORIGINAL LIT, MORE 


SELECnVELY ^ 

Low 


4o 


6o 


387 


Medium 


39 


6l 


155 


-High 


47 


53 


99 


ALLOCATING R&D FUNDS MORE 
EFFECTIVELY . 


LOW 


40 


60 


599 


Medium 


54 


46 


28 


High 


57 


43 


14 



ERIC 



Figure 24. Extent of Reading and Skimming, According to 
Purposes Served by the Annual Review 
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Use of the Annual Review for "exchanging information with your colleagues" 
received the most responses in the "often" category of Q^iestion l4. These 
results are intriguing, because they suggest that the Review, which is an 
inroersonal information channel, may also be used as an adjunct to interpersonal 
communic^ion. Other activities frequently associated with use of the Review 
are writing reports, articles, and papers, and ordering documents. 



In the future, for which purposes do you 
think you may use the Annual Review? * 


Seen/Heard 


keeping up with current work in your 
own areas of interest 


m 


keeping up with current work in 
peripheral areas of interest 


46 


checking on particular projects, 
ideas, or activities of individuals 


24 


other 


4 



Multiple responses were permitted, so totals will not 
equal 100^. 



Figure 25. Formatted Qnestion to Seen/Heard Group 
on Anticipated Use of Annual Review 
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12 Have you obtained from the Annual Review itself information about 
any of the following?* 



ideas or theories 51% 

methods or procedures 50 

results or data 2k 

an individual and his work 22 

a system 31 

a project 26 

' other ' k 



l4. How often have you used the information you've obtained from the 
Annual Review for each of the following activities? 





Often 


Some- 

times 


Never (+ NR) 


Total 


Ordering documents or 
publications 


6% 


42% 


52% 


100% 


Searching the literature as a 
service to others 


3 


29 . 


68 


100 


Writing reports, articles and 
papers 


9 


43 


48 


100 


Preparing lectures or speeches 


8 


27 


65 


100 


Writing your own review of 
the literature 


4 


12 


84 


100 


Preparing a bibliography or 
reading list 


8 


33 


59 


100 


Exchanging information with 
your colleagues 


10 


46 


44 


100 


Other 


3 


2 


95 


100 



Multiple responses permitted, so totals will not equal 100%. 



Figure 26. Additional Questions on Purposes Served by the Annual Review 




i*. , w I , ‘ •* •— 



15 November 1968 69 TM-4125 



5.5 CORRELATES OF PREVIOUS CONTACT WITH ANNUAL REVIEW-CITED LITERATURE 

The variable "previous contact with cited publications," representing 
familiarity with the publications cited in the Annual Review before readiiOR 
the Annual Review, is the sole indicator of a respondent's relationship to 
the book's content prior to publication of Volume 1. Thus, it is the only 
measure that can be used with any of the other variables to indicate the 
impact, or changes, brought about by the Annual Review. (In contrast, there 
are five measures of post- reading behavior, to be reported below. ) 



5.5«.l Pri mary Findings 

Only 8^ of the respondents to Question 5, below, had had "practically no" 
contact with the cited publications; the rest of the respondents were able 
to claim at least "some" contact (53^)> if not more exhaustive contact 
(38#). One can say, therefore, that 38# of the readers had had "high" 
previous contact, and 62# "low" previous contact. 



5. About how many of the publications cited in the 
chapters within your particular areas of interest 
had you read before you saw them cited in the 
Annual Review? 


NR 


1# 




practically none 


8 




some 


53 




quite a few 


33 




most 


Total 100# 







Figure 27. Previous Contact with Annual Review-Cited Literature 
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It is difficult to place any great confidence in the particular numbers ob- 
tained. They could be the result of a poorly worded question that did not 
adequately discriminate among respondents. The meaning of "some" contact 
is vague, and it is a very easy response to mark on the questionnaire. On 
the other hand, the question could be a good indirect measure and internal 
data check on the function of the Annual Review. If over half the readers 
in this survey actually were not very well acquainted with the current 
publications in the field of information science before reading the Annual 
Review, then their primary use of the Annual Review to "keep up" with the 
literature, as shown by other data, is understandable. 

The desirability of a reader’s having had extensive acquaintance with the 
publications cited in the Annual Review is another issue, probably dependent 
on the purposes for which it is read. Analysis of the data shows that the 
extent of previous contact is significantly related to only one purpose of 
reading the Annual Review: keeping up in one's own area of interest. Those 

with high previous contact tend to find the Annual Review much less useful 
for keeping up than those with low previous contact, although this 
tendency decreases with increasing number of chapters read. That is, the 
more chapters read, the more useful is the Annual Review for current 
awareness in one's own are a- - even with high previous contact. This is 
probably a manifestation of the value of the critique of the literature 
that the Annual Review provides. The role of extensive previous acquaint- 
ance with the cited publications is probably also a function of the extent 
to which Annual Review chapter authors offer brief, evaluative (rather than 
reportorial) comments about the publications. If the literature in infor- 
mation science keeps expanding to the point where some of the studies of 
moderate importance cannot even be mentioned (for one reason or another) 
by Annual Review authors, and in addition the authors write very briefly 
about each one of the very important studies or developments, it may become 
vital for readers to have extensive previous contact with the publications 
cited, in order for them to benefit at all from the booki This, in turn, 
limits the effectiveness of the Annual Review as a teaching or orientation 
tool. 

While the worth and function of high previous contact with cited publications 
may not yet be evident, it is certainly apparent that previous contact does 
not serve as a deterrent to extensive reading and skimming. Figure 28 
shows that those with low previous contact with cited publications have a 
strong tendency to be low on reading/skimming behavior, while those with 
high previous contact are as likely to be low as they are to be high on the 
index. Although this can be interpreted as an art i factual correlation 
between the "aided recall" provided by reading the Annual Review and 
recognizing cited publications, it is equally likely that the apparent 
consequences of high previous contact could be explained by personal 
attributes such as job activities, information-use activities (such 
as journal readership), or normal informat ion- source patterns. These are 
discussed in the next section. 
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Previous 


Contact with 




Cited Publications 


Read/Skim Index 


Low 


High 


Low 


62% 


49% 


High 




‘ ^ 


Totals 


100% 


100% 


N = 


4oo 


24l 



Figure 28. Annual Review Readership In Relation to Previous 
Contact with Cited Publications 



5«5‘2 Relationships between Previous Literature Contact and Other 
Personal Attributes 



There Is no change in the previously repotted tendency for Users with little 
previous literature contact to be low on Annual Review chapter readership, 
as measured by the Read/Skim Index, when the six Job activities that are 
best represented among Users are considered.* However, for Users with "high” 
previous contact, the 49-51 split on the Read/Skira Index does change for 
some activities. Being involved primarily in library reference activities 
seems to make a difference; the split is 57-43. For those with high previous 
contact, those naming "research” activities also tend to be significantly 
lower on readership than those not in research. This is not too surprising. 
Researcher personnel tend to be rather specialized in their interests, and 
it is possible that no more than two or three chapters in the Annual Review 
will be of direct interest to active researchers. 



Hypotheses regarding a relationship between two variables are tested by the 
addition of a third, or control, variable to the cross-tabulation. If the 
basic direction of the relationship changes substantially, then the rela- 
tionship is said to be spurious. If the trends in the original table re- 
main the same, then the relationship is not spurious, and the hypothesis is 
not rejected. 
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Being involved in design/development, management and administration, and 
consulting does not seem to affect the original relationship between "high" 
previous contact and readership. Finally, writers and editors with high 
previous contact are significantly higher readers/skimmers. An explanation 
for this might lie in the nature of the activity itself: writers may be 
active readers by nature. 

There is also a relationship between previous contact with cited publica- 
tions and various kinds of information-use activities. Attendance at 
three or more professional meetings, and presentation of even one paper at 
a meeting, pushes those with high previous contact in the. direction of 
high reading/skimming behavior. Even though these data are significant 
by the chi-square test, they do not support an inference of any causal 
relationship. Since we have no actual knowledge of the time order of these 
three variables, the use of reading/ skimming behavior as the dependent 
variable is arbitrary. Nevertheless, the three do provide an interesting 
correlation, inteipretable as a high general activity level in these 
respondents' information-use behavior. 

The relationship between previous contact and reading/skimming level seems 
to be a function of the particular information sources involved. Of the 
Users with high previous contact who consider colleagues within their 
organizations and professional meetings to be "very important" information 
sources, slightly more than half are high on the Read/Skim Index. Other 
study data would have led one to predict almost a 50-50 qplit. Although these 
changes suggest that interpersonal communication is a factor in prompting 
one to read the Annual Review, they are really too slight to be given much 
importance. A slight and equally insignificant change in the opposite 
direction is evident for Users who consider preprints to be very important 
information sources. Their high previous contact with cited publications, 
presumably as a function of their residing of preprints, tends to make only 
43^^. of them high on the Index, instead of 51^. Similar small changes were evi- 
dent for Users with low previous contact for whom technical reports, reprints, 
and abstracting or indexing services are inportant information sources. Again, 
these changes would need further analysis to explain their significance 
(if any) for understanding the original relationship between previous 
contact and reading/skimming. 
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5.6 RELATIONSHIP OF ANNUAL REVIEW READERSHIP TO OTHER USER 

CHARACTERISTICS 

5.6.1 Number of Years in Information Science 

As In^cated earlier (Section 5.1.2), all of the questionnaires asked "How 
many yeeurs have you worked in the field of information science— i.e. , 
documentation, library science, information systems research, or related 
fields?" The resulting data for Users were also shown earlier, in Figure 11. 
Since the median of the response distribution falls very close to the ten- 
year point, the distribution was divided at that point. To investigate 
the relationship between this "professional age" variable and the Read/Skim 
Index, the two were cross-tabulated. The results, which are shown in 
Figure 29, indicate that Users who had worked in the information science 
field for less than 10 years tended to use the book less than those who 
had 'Jeen in the field longer. 







Number of Years 


in Infomatlon Science 








<10 


> 10 


Read/Sklm 

Index 


Low 1 










Hlgh| 




,JSjo .. 


ji5i 






Totals 


1005^ 


looi 






N - 


3*t3 


298 



Figure 29. Relationship between Annual Review Readership 
and Professional Age 



There are several possible explanations for this finding. One is that new- 
comers to information science do not yet have the breadth of Interest and 
experience to warrant reading (or skimming) of many different chapters. 
Because of the way the Annual Review is prepared— with a fairly experienced 
professional audience in mind — newcomers are unlikely to be equipped to use 
it effectively. Another related possibility is that those with more experi- 
ence in the field may have roots in the peripheral and parent fields from 
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which information science has grown and, therefore, need the critical- review 
or orientation-to-new-literature aspects of the Annual Review to keep up in 
their specialties. Additional analysis supports the latter explanation: 
the majority of Users with more experience who found the Review "very useful" 
for keeping iqp with the literature are indeed high on the Read/Skim Index, 
i.e., they read a great deal of the Review. And of the respondents with 
more experience and who valued the book for orientation or learning about 
peripheral areas, 70fo were high on the Index— a considerable increase from 
the that would be expected on the basis of the results shown in Fig- 
ure 29 . 



5 . 6.2 Primary Job Activities 

To investigate User characteristics that might influence readership of the 
Review, a cross-tabulation was made between the Read/Sklm Index and the 
five job activities with the highest response percentages (management/ 
administration, design/development, research, library reference, and 
consulting). Only two significant differences among readers who spend any 
portion of their time in the five activities were found. Those in 
consulting tend to be heavy users of the Annual Review, while those in 
library reference work tend to use the Annual Review less than average. 

Both of these trends seem easy to explain. Consultants have to read 
extensively to maintain their coii5)etence in any given field, while 
librarians, whose daily work is not "information science," may find that 
the Review is too "academic" for their purposes and are therefore discouraged 
even from skimming it. 



5 . 6.3 Importance of the Annual Review as an Information Source 

The goal of this analysis was to determine how important the Annual Review 
is to different kinds of readers. Figure 30 is the result of cross- 
tabulating the Read/Skim Index by one part of question I 5 for Users, the 
importance of the Annual Review as an information source. The figure 
shows that importance is positively related to readership. The figure 
could be interpreted in two ways: the greater the perceived importance of 

the Annual Review, the more it is read; or, the more it is read, the more 

important it becomes to the readers. The data provide much greater support 
for the latter interpretation. The experience of Users could have discon- 
firmed the expectation that the Annual Review would be valuable, but it did 
not. 
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Importance of the Annual Review 
as an Information Source 



Read/ Skim Index 


Very 

Imoortant 


Moderately 

Important 


Not 

Important 


Totals 


N g 


Low 




(>zf, 


20^ 


100^ 


380 




32 


53 


15 


100 


261 



Figure 30. Relationship between Annual Review Readership and 
Judged V6Llue of the Review as an Information Source 



^.7 IMPACT OF THE AHNUAL REVIEW 



As indicated earlier, it is necessary to obtain data over a period of time 
to detennine what impact a particular technique or tool has had on 
professional conmuni cation. In a sense, all of the data obtained in the 
present survey serve primarily as baseline data against which later data 
can be conpared, to provide a true measure of impact. Nevertheless, some 
conclusions about the impact of the Annual Review can be drawn on the basis 
of the questionnaire items. 

One way to measure the impact of the Annued Review is to note the information 
seeking behavior it prompts, i^aestions 6 through 10, shown in Figures 31 
and 32, sought to assess the Users' behavior toward the publications cited 
in the Annual Review. Over 9Cff> of the Users reported (Qaestion 5, Section 
5.5) at least "some" contact with the publications cited before they read 
the book; yet more than half examined some of these publications again, 
as shown by the data from Question 6 in Figure 31. Various reasons were indi 
cated (Question 7, Figure 31, below), the most frequent being simply to "re- 
view/ details." It is an interesting and inportant finding that 20^ of those 
who re-extamine a publication as a result of reading the Annued Review do so 
to re-evaluate it in the light of an Annual Review author's comments. Al- 
though one cannot prove, from the data, that the re-evaluation was rewarding 
and productive, this is a reasonable supposition. Some support for this 
belief comes from the responses to Question 8. Over half of the Users tried 
to obtain Annual Review-cited publications that they had not seen before. 

This would certainly seem to indicate acceptance of the Annual Review chap- 
ter authors as authoritative guiLdes to valuable literature. 
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6. 


Did you examine any of these publications again as 


a result 




of reading the Annual Review? 










Yes: About how many did you examine again? 






1 - 5 


29i> 








6 - IvO 


15 








11 - 15 


5 








l6 - 20 


2 








21 - 30 


1 








mor e tiran 30 


1 




< 




No, NR, none 


hi 








Total 


100f> 






7. 


If you have examined any of the publication»s again, what 




were your reasons for examining them? To: 










Yes 


No 


Totals 




review details you'd forgotten 




70^ 


ioaf> 




re-evaluate a publication in 


20 


8o 


100 




light of an Annual Review 
author's comments 










cite a publication or use the 


19 


81 


100 




information in a report, re- 
view, or article of your own 










obtain information on other 


5 


95 


100 




works by a publication's 
author 










other 


3 


97 


100 


8. 


After reading the Annual Review, 


have you 


tried to 


obtain 




some of the publications cited in it that 


you hadn 


't read 




before? 










Yes: About how many? 










1 - 5 


31^ 








6-10 


15 








11 - 15 


5 








l6 - 20 


2 








21 - 30 


1 








more than 30 


1 








No, NR, none 


h3 








Total 


ioof> 







Figure 31. Behavior Toward Publications Cited in the 
Annual Review 
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9. i^s a result of reading the Annual Review, have you tried 
to contact any of the authors mentioned for information 
regarding their current work? 

Yes 10^ 

No, NR 90% 

10. Has t'ne Annual Review Itself ever suggested to you 

specific Ideas for future research projects or studies? 

Yes 45% 

No 55 % 



Figure 32. Additional Effects of Annuel Review Readership 



Additional corroboration of the stimulative effect of the Annual Review 
comes in the responses to Questions 9 10 (Figure 32). Ten percent of 

the Users tried to contact cited authors, as a result of reading the Annual 
Review. This seems like a rather remarkable amount of personal contact for 
a publication to trigger. It would be most Interesting to see how this 
figure changes over time. 

The responses to Question 10 also provide something of a surprise, with 45% 
of the Users reporting that the Annual Review suggested to them specific 
ideas for future research projects or studies. It would be of some Interest 
to learn whether reading the Annual Review suppressed some previously 
developed research Ideas by indicating either tnat the problem involved 
was already being worked upon e^enslvely or that it was much more difficult 
than the reader had supposed. Answers to such questions must await another 
survey opportunity. 

Some other interesting relationships are apparent from the data. The 
analysis summarized in Figure 33 shows that those with high previous contact 
with cited publications have a greater tendency to re-examine publications 
and to contact mentioned authors than do those with low previous contact. 

On the other hand, they are somewhat less likely to get specific ideas for 
future research. The difference is small (only five percentage points), 
but It is not in the direction one might expect. 
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Post-Reading Behavior 



Re-Examine Seek New^ Contact Contemplate 
Publications Publications Authors Future Research 

Previous Contact 
with 

Cited Publications 



Low 




59f> 




>*Tf> 


Ns4oo 


High 


6l 


55 


12 


42 


Ns241 



Figure 33 » Amount of Previous Contact with Cited Publications 
Among those Active in Post -Reading Behavior 



Figure 34 elaborates these results by presenting some particularly clearcut 
and revealing trends for users at different levels of readership. The 
conclusions are that as total readership increases, so does every form of 
post-reading behavior. This strong correlation of reading with important 
post-reading behavior suggests that the Annual Review, even in its first 
volume, had fulfilled one of its goals: to stimulate productive activity 

in information science. 
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Post«Readlng Behavior 



Chapters Read 
in Detail 


Re-examine 

Publications 


Seek New 
Publications 


Contact 

Authors 


Contemplate 

Future 

Research 




Low 


(0-2 chapters) 


39f> 






35jt 


N»266 


Medium 


(3-5 chapters) 


57 


62 


9 


52 


N-215 


High 


(6-12 chapters) 
Read/Sklm Index 


69 


73 


l6 


52 

(Total Users 


n«i6o 
«= 641) 


Low 


46 


47 


8 


39 


N»38o 


High 


62 


65 


11 


54 


N«261 



(Total Users =* 64l) 



Figure 34. Readership Among those Active In 
Post-Reading Behavior 



5.8 USER SUGGESTI(»fS FOR IMPROViMENT 

5*8.1 Future Changes In the Annual Revlev 



Users had several opportunities to offer suggestions for changes and 
Improvement In the Revlev. Two questionnaire Items ^ shown In Figure 
35) Invited comments. 
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l6. If one chapter of the Annual Review were to be published 
each month Instead of collecting the chapters In book 
form annually > how do you think the value of the Annual 
Review would change? A different chapti^r each month 
would be: 



less useful 

equally useful, multiple response 
more useful 


52$ 

17 

25 


NR 


6 


Total 


lOO^t 


WHY? 




In the future, what aspects of the Annual Review might 
be changed to make It more useful? 


(This question was coded separately, 
percentage = 34^) 


Response 



Figure 35. Questions Eliciting User Suggestions 
for Improvement 



Answers to the free-response portion of Question l6 were not classified 
Into different categories^ because the question apparently was ambiguous 
to some readers. (See Section’ k,k for discussion of the coding practice 
adopted for da’ta analysis.) However, the responses are still of some 
value. 

Reasons given for thinking that a monthly chapter would be more useful were 
that the cliapters would be easier to file and retrieve later and that 
they would save reading time (both of these were also listed as reasons 
for preferring an annual volumel); that It would be easier to absorb the 
Information In a monthly chapter; and that a monthly chapter would be more 
current and would reduce the publication lag. 



15 November 1968 



81 



TM-4125 



Respondents who preferred the annual volume cited its usefulness as a 
bound, collective review that surveys the entire state-of-the-art and 
that reassures them that they have missed nothing important in their 
areas of interest during the year. These respondents also appreciated 
the fact that the Annual Review is an edited review, with index and 
bibliographies, and that, as a bound volume; it is more permanent. 

The response rate for Question 17 was or 220 responses. Figure 36 
shows the response categories, which were developed by inspecting all the 
answers and then classifying the responses into the six classes shown In 
the figure. 

Some of the ideas expressed in these suggestions had already been taken 
into account in the preparation of Volume 2, which was published 
approximately one month after the survey was initiated. Others have been 
Implemented in later volumes. For example, regarding style of writing, 
authors are regularly encouraged by the Editor to make their descriptions 
of studies informative, rather than merely indicative, and to supplement 
their descriptions with an appraisal of the significance of the work 
described and its relationship to other work. This guidance is re- 
inforced during the process of reviewing and criticizing the first-draft 
manuscripts . 

With respect to content, each volume has seen changes in coverage, with 
an eye to providing a better mapping of the domain of information science. 
For example, in lieu of a broad-brush treatment of "Information Systems 
Applications" in Volume 1, separate chapters have appeared In later 
volumes on applications in chemical documentation, in medicine, and In 
education. 

The interest in "better organization" is comparatively strong but not 
easy to respond to. Organization has been Improved in Volumes 2, and 
4 by capitalizing on the experience with Volume 1 to make clearer 
divisions among the chapters and to place them in the most logical sequence. 
Some effort has also been made to provide coordination among the chapters, 
although their nearly simultaneous completion each April precludes any 
really detailed cross-chapter comparisons. 

Access tools have been improved since Volume 1. More sources of 
publications are given and plans are under way to develop a cumulative 
index to Volumes 1 through 4. No plans have been made, however, to in- 
clude price information in the chapter references, since this information 
is not uniformly available to the authors. 

The publication cycle has been shortened; Volume 3> covering all of 
calendar year 19^7 > available before September 20, one month before 
the annual ASIS convention. The third volume also Introduced a new type- 
face, designed especially for the Annual Review series. 
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Suggestion 


Frequency 
of Mentioning 


Style of Writing 




1. Authors should be more critical 




2. Authors should give more detailed information, e.g., 
results of studies 


12 


Content of Volume 




3 . Review should reflect all that is going on in infor- 
mation science (even though information science is 
still in a formative stage) 


4 


4. Specific chapters (named) should be added 


16 


5» Specific chapters (named) should be deleted 


1 


6 » Review should Include a summary of work in progress 


3 


Organization of Volume 




7» Material should be better organized 


12 


8 . There should be better coordination the 

chapters, with more comparison 


14 


Tools for Access 




9 . Sources of cited publications should be listed 


2 


10. Prices of cited publications should be given 


1 


11. Provide new indexes, appendices, combined bibliography 


7 


Mechanics of Production 




12. Publication cycle should be changed, e.g., so book 
would be available before the ASIS convention 


10 


13* .Typography should be improved 


2 


Miscellaneous 




l4. Price of the Review should be changed 


2 


15 . (Compliments, e.g.,"l like it the way it is”) 


7 


— >• Total 


lOO^ N =220 




Figure 36 . Suggestions for Improving the Annual Review 
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Ihe price of the Annual Review is not subject to editorial control, 
except in the sense of limiting the number of pages that will be permitted 
for each volume. Fairly strict page limitations were used for Volume 3, 
to meet a maximum acceptable price agreed upon between ASIS and the 
publisher. 

Survey respondents and ASIS members had additional opportunities to 
provide feedback on the Review at informal group discussion sessions and 
in personal interviews. Group discussions and personal interviews were 
held with about 20 Users at the October I967 ADI Annual Meeting in New 
York, and at a meeting in Chicago at Encyclopaedia Britannica, also in 
October I967. These Users were invited to indicate how they used the 
Review and to offer suggestions for improving it. The uses of the Review 
listed by respondents in this small subsample were essentially the same 
as the uses listed on the questionnaire. The suggestions from these 
discussions for Improving the Review include making the cited publications 
more easily available and including more detailed information (such as 
study results) in the chapters. 

A ’’Feedback Forum" with the Editor of the Review and staff members was 
held at the 1968 ASIS Annual Meeting in Columbus, Ohio, and numerous 
suggestions for improvement were discussed. 

One person was concenied over the relatively weak coverage of the 
secondary services in Volume 3 and the planned Volume 4. She conceded 
that secondary services and facilities were touched upon in several 
chapters of Volume 3 but felt that it would be more desirable to cover 
them in their own right, even on a bi-annual basis, than to "bury" the 
mentions in various other chapters.* 

One person questioned whether an annual review was necessary in the 
information science field. She argued that advancement is very slow and 
that there are few "real” developments. Several members of the audience 
commented that, even if true, this situation would not necessarily reduce 
the need for a synthesis of what was being done in the field. The same 
person also went to considerable pains to describe the difference between 
the highly selective, critical, penetrating type of review aimed at only 
the "front-of-the-wave" workers, and the more descriptive, superficial, 
tutorial review with wider coverage of material. She was most interested 
in the first kind and thought it a waste of time to try to get exhaustive 
coverage. There was dissenting opinion from the floor, and the chair 



A subsequent outcome of the meeting was the decision to add a short 
chapter on the Secondary Services to the current Volume (4). 
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pointed out several other use goals of the Annuel Review and cited the 
survey results to indicate that the Annual Review served many gosils for 
many people. Someone suggested that the Review should Include ^»th 
critical and tutorial chapters^ and this was ignored^ by all handls. 

There was also considerable discussion on the ov€>rlap in coverage; among 
the chapters. One person was annoyed by the oves’jLap; others felt; that 
it was inevitable. One person suggested that the Editor would be derelict 
if all overlap were eliminated^ presumably becauee very few persons read 
every chapter of the Annual Review. Thus^ each chapter has to be 
relatively self-contained^ even at the risk of some overlap with other 
chapters. 

There was considerable sympathy for the Idea^ expressed by one member of 
the audience^ that adding some kind of overview chapter would Improve 
the Annual Reviews. While a brief Introductory statement preceding each 
chapter might provide part of the inter-chapter linkage that appeared 
desirable to the audience^ more of them felt that a single chapter would 
be preferable. The Editor pointed out the inqsll cations for the production 
schedule of adding a final overview chapter^ to be prepared after all of 
the other chapters were completed^ and it seemed clear that most members 
of the audience would not be as much in favor of an overview if it meant 
introducing a substantial delay in the availability of each volume. The 
suggestion that the overview of Volume 3 appear in Volume 4 did not meet 
with much enthusiasm. 

A related suggestion^ ardently advocated by one member of the audience^ 
was for the inclusion of "rapporteurs" for each chapter. These would be 
high-status persons (persons who could "speak with authority from another 
viewpoint") who would be asked to do short papers^ to appear at the ends 
of each chapter. There would be no opportunity for rebuttal from the 
author. Two or three former chapter authors Joined in vigorously to 
object to the Idea^ one in particular indicating that this procedure would 
be an excellent way to dry up the supply of chapter authors. The Editor 
pointed out the scheduling difficulties this suggestion would impose^ 
indicating it would not be* possible to maintain the present production 
schedule^ which is the shortest he is aware of for an annual review. 

Other suggestions Included using previous authors as additional reviewers 
for current chapters; providii^ the authors with on-line editing 
capability; putting the chapter references at the end of the volume, 
presumably to eliminate duplication of entries; and adopting "more 
flexible scheduling," so that chapters would not be planned until it was 
known what literature had been produced during a given calendar year. 

All of the suggestions will be discussed with the ASIS Advisory Committee, 
for their possible value in planning future Annual Reviews. 
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5-,3.2 Nonforma ttable Responses In the ” Other" Categories 

A di.fferent form of feedback was the non-structured responses In the "Other" 
category of several questions. The incidence of these responses was very 
small^ and almost all of them could have been classified into the 
structured categories. Most responses that could not be so classified 
reflected a misinterpretation of the question. The "Other" categories of 
Questions 13 and l4^ discussed earlier., in Section provide examples 
of this misinterpretation. Questions 13 and l4 attempted to separate 
purposes for using the Annual Review from activities associated with using 
the Review. Both questions provided an "Other" category. The responses 
in this category for both questions^ however^ seemed to confuse purpose 
and activities. For instance, "learning" and "general background" were 
two responses to Question l4, which asked about activities associated with 
use of the Review. Some amount of such confusion Is probably an inevitable 
result of any initial survey, and it is fortunate that there were few 
responses in the "Other" categories in this survey. 

A rather unusual response in the "Other" category for Question 13 was use 
of the Review "to assess the competition." If this response had had to 
be categorized. It could have been placed in the "current awareness" 
category, but It is much more intriguing Just by Itself. This response 
suggests that the Review, which Is an Impersonal information channel, 
was used, in part, as a substitute for an interpersonal channel, such as 
professional meetings, which are often used to assess the activities of 
competing organizations. 
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6 



DISCUSSION 



6.1 



ROLE AND IMPACT OF THE ANNUAL REVIEW 



Many previous studies of professional communication have pointed to the impor- 
tance and high use of review literature. The results of the present study, 
dealing with a particular form of review literature — an annual review — confirm 
and extend the previous findings. Readers of the Annual Review of Information 
Science and Technology find it useful for many different functions, this in 
itself makes it a rather unusual information tool. A textbook provides orien- 
tation to a field; a journal article usuaily conveys information about a 
particular new concept or activity; bibliographies, particularly selective 
ones, point to useful literature. The Annual Review serves these and other 
equally Important functions — particularly, critical evaluation — in a single 
pacl^ge . 

Evidence has been described that Users of the Annuai Review engaged in such 
post-reading behavior as reexamining cited publications that they had already 
seen, seeking new publications cited, and attempting to contact authors cited 
in the Annual Review chapters. Users also reporting getting new ideas for 
research as a result of reading the Annual Review. All this is clear evidence 
of extensive Impact. The fact that the Annual Review does not appear to reduce 
the use of other information channels — Users have not given up reading the 
primary, literature — is not a deficiency but an asset. Evidence from user 
studies Indicates that, except in some rare Instances, use by scientists of 
one Information channel does not displace use of other channels. Rather, the 
greater use of one channel leads to greater use of the other channels, too. 

The ERIC system, which one might suppose is in some way subversive to journal 
use, may actually enhance it. All information channels have something of a 
"synergistic" effect--they interact and mutually reinforce each other. This 
effect seems to be particularly strong for those channels, such as annual 
reviews, that point toward other information channels. The Annual Review 
seems to stimulate, not displace use of these channels. 

This study found the impact of the Annual Review to be high. Now, it was 
mentioned earlier that persons who are aware that their behavior is being 
observed, as they were in this survey, tend to modify it — or to modify their 
reports about it — to accord with their surmises about an experimenter’s motives 
and values. Since their surmises may have colored some of their answers. It is 
possible that the high impact found should be partially discounted. On the 
other hand, the time available for the Annual Review to even have been heard of 
before the survey was relatively short; that it was rather widely known, and 
that those who reported using it found it as useful, for as many purposes, as 
they did, suggest that any Inflation In the answers to the questionnaire is 
probably offset by the brief exposure on which the survey was based. It can be 
concluded, therefore, that the Annual Review's impact was In fact not only high, 
but remarkably swift. Beyond that, it can be assumed that Its impact Is now 
much greater than it was at the time of the survey, not only because of the 
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greater time that has elapsed, but also because the three volumes that now 
exist support a reference use that was necessarily limited when only one volume 
was available. 

There is another reason to assume that the Impact of the Annual Review has 
grown. Those readers who made suggestions for Improvements touched upon a 
number of changes that, in their opinion, would make the Annual Review more 
useful. Many of those changes have been made, and it is reasonable to suppose 
that they have helped to increase both effectiveness and impact. 

Some of the changes that were suggv'.sted, such as that of making the Annual 
Review more tutorial are, in effect, suggestions to deliberately change the 
type of impact the Aixnual Review seems to have had and to make it effective 
for quite another audience than the one that seems to have found it effective 
thus far. These changes, of course, involve some serious risks. If one were 
to draw a curve representing the professional-activity level, or the contri- 
bution of various information of scientists and specialists, the distribution 
would in all likelihood be skewed. There would be a very few highly active 
and Influential contributors, a large number of somewhat less active contri- 
butors, and a still larger number of even less active contributors. Finally, 
there would be a very large number whose contributions to the field, because 
of their limited experience, opportunity, or talent, are quite small. The 
latter group is the one that might benefit most from a tutorially oriented 
annual review publication, and thus it is tempting to adopt such an orientation 
not only to serve an educational goal but also, perhaps, to tap a broader 
market. One needs to say "perhaps" because some user studies suggest that a 
not insignificant proportion of the largest group displays "information apathy" 
because of their work situation or personal goals, they do not seek out infor- 
mation and they tend to ignore it even when it is readily available. Even if 
a much less technical and sophisticated treatment — one, in fact, that would be 
of very limited use to the smaller group of energetic and conscientious pro- 
fessionals whose activities are most likely to help accelerate the growth and 
advancement of the Information science field. 

One way to minimize the market-limiting effect of a sophisticated, or fairly 
sophisticated;, annual review treatment is to give close attention to what are 
termed the presentation attributes of the book, l,e,, the attributes other than 
content. One can, by careful choice of formats, typeface, graphic displays, 
and even paper stock, increase the attractiveness and Inviting qualities of a 
book without compromising the quality of its technical content. It Is also 
possible to Increase the Inviting qualities of the book by Including an intro- 
ductory chapter that, by previewing highlights or summarizing major trends, 
encourages further reading of the book. An Introductory chapter need hot, of 
course, alter the content of the remainder of the book, A number of readers 
have suggested such a step, and it is under consideration. 
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6.2 METHODOLOGIC/iL II-IPLICATIONS OF THE STUDY 

The point has already been made that the present study was concerned with both 
impact and effectiveness. One reason for not limiting the study to impact 
measures was that there were very few such measures that one could hope to 
obtain with the very narrow time base involved in this study — less than ont 
year. It is, in other words, too early for an accurate assessment to be made 
of the Impact of the Annual Review on professional communication. 

A second, equally important, reason for being concerned both with impact and 
effectiveness is that the two are interrelated and may interact in several 
important ways. Consider, for eaiample, a new text that Is highly readable, 
well organized, and attractive, but that is technically glib and superficial. 
Such a text may drive texts of much higher technical quality right out of the 
market. In such a situation, one would consider the new text to have high 
impact but poor effectiveness, if we define effectiveness in terms of advanc- 
ing the field of inquiry. On the other hand, there may be some publications 
that are highly effective but have little impact because their limited sales 
make them very difficult to obtain and use. 

The ideal kind of appraisal of a communication tool is one in which the effec- 
tiveness of the tool itself is examined before examining its impact on other 
communication tools or channels. Such an appraisal procedure helps to ensure 
that judgments about the actual or potential impact of the tool take into 
account the distinctive characteristics of the tool being assessed. Some 
annual reviews severely discourage the citation of unpublished reports; others, 
including the Annual Review of Information Science and Technology , encourage 
their use, to provide a richer and more current literature base for the analysis 
of progress. The Annual Review also gives painstaking attention to technical 
and bibliographic accuracy, through a complex reviewing mechanism; some other 
annual reviews consider the quality of the chapter contributions to be entirely 
the concern of the authors and accept the contributions very much "as is." 
Obviously, one should know the characteristics of the particular annual review 
publication being studied, to interpret or generalize findings regarding its 
impact. 

The data from this survey are not only rich with respect to the impact of 
Volume 1 of the Annual Review; they also provide a good baseline for future 
studies of the Review’s impact. This first study provides a valuable "before" 
measure of impact, so that changes over time can be determined by periodic 
measurements. Because the data are so rich, it should later be possible to 
perform a number of secondary analyses, i.e., additional analyses of the same 
variables, but with a different focus. The new publishers of the Annual Review, 
Encyclopaedia Britannica, have used pre-publication order forms for Volume 3 
that also offer Volumes 1 and 2 to the buyer. By matching the names and 
business affiliations of people who have ordered Volume 3 with the names and 
business affiliations of survey respondents, one can identify appropriate 



15 November 1968 



89 



TM-4125 



samples on which various other kinds of pre- and post-reading behavior can be 
examined . 

From a methodological point of view, the survey was very successful. The 
response rate (67%) was surprisingly high. The size of the sample, and the 
response rate, permitted a sound analysis of the data. Each of the three 
groups was large enough that the results of the cross-tabulations could show 
trends clearly. The comparison of Users, Seen/Heards, and Not-Seen/Heards 
proved to be \i useful design for the study because the latter two groups could 
serve as control groups. And although the findings of the survey cannot be 
generalized to apply to all information scientists, they are likely to be true 
for most members of ASIS. 

TJhen the study is repeated, one will certainly wish to make some changes in 
the questionnaire. The questions that permitted free responses might be modi- 
fied to make them easier to code. For example, the critical-incident question 
("For what purpose did you [last] use the Annual Review?") might be re-worded 
so that the environment and activity associated with the last use of the book 
might also be determined. Also, a follow-up question to the critical**incident 
question might probe further the browsing use of the Review that emerged from 
this question. Browsing is an important adjunct of information-seeking and 
information use, and interesting and/or useful information is often found by 
browsing through a book like the Review. 

The rate of non-response for each question varied widely. This may have been 
due to laziness, impatience with the questionnaire, misunderstanding of the 
question, lack of information, or lapse of memory. The questions that elicited 
a high non-response rate in this, study should either be deleted from future 
studies, or re-worded. 

It would also appear desirable, in the future, to adopt somewhat less obtrusive 
measurement procedures than were involved in the present study, so that the 
risk of having respondents second-guessing the questionnaire can be avoided. 

The way to avoid it is to use less obtrusive measurement procedures, such as 
counting sales, citations, and/or instances of library use of the Annual Review. 

The questionnaire data were collected in the Fall of 1967. It. seems highly 
desirable to schedule the first follow-up for the Fall of 1969, after the fourth 
volume in the series is published. The sample should Include some of the same 
persons — Users and non-Users— involved in the initial survey; it should also 
involve a number of persons not previously included in the survey. Whatever the 
experimental design, the data initially obtained should be regarded as a valuable 
resource in a thus far unique inquiry into the role of annual reviews in pro- 
fessional communication. 
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6.3 IMPLICATIONS FOR SPONSORSHIP OF ANNUAL REVIEWS 

The general finding that review publications are widely and increasingly useful, 
and the finding from this study that an annual review publication can be a 
powerful stimulant to professional communication, have an important implication 
for professional associations, publishers, and organizations that sponsor 
scientific and technical research. It is that annual reviews may have enormous 
potential for advancing the field of inquiry and should, where possible, be 
encouraged and supported. 

When is an annual review desirable? Presumably when the literature of a field 
has grown to such a size that it is difficult or impossible for most of the 
active and conscientious professionals to keep fully in touch with developments. 
In practice this condition is not easy to identify because the characteristics 
of a "field" are not well understood. One cannot easily tell whether a given 
range of activities constitutes a single field or clusters of unrelated sub- 
fields. Some clues can be gained by such techniques as examining the inter- 
citations among the journals supported in a "field"; the more the inter- 
citation, the mere one is justified in considering the activities as repre- 
senting a single field. Yet, an annual review may itself be able to accelerate 
the development of a cohesive field from fragmentary subfields by facilitating 
contact with the literature of other subfields and, thus, encourage the recogni- 
tion of related activities in these subfields. There is some reason to believe 
that the Annual Review may be having some such effect in the field of informa- 
tion science, which, only a few years ago, was considered a field by many fewer 
people than the number that are now willing to regard it as such. 

In spite of the difficulty of providing any precise answer to the questions, 
"When is an annual review desirable?", there are some situations in which the 
decision that it is desirable is justified. The most obvious is that in which 
a large number of professionals — perhaps upwards of 3, 000- -are not served by 
any existing annual review. Using this standard, there are obviously several 
fields that are overripe for an annual revievy (or for more annual reviews) . 
Education is a prime example. On the other hand, fields or areas such as com- 
putational linguistics arc not ready, because the practitioners are few, the 
literature is sparse and, as yet, manageable, and the rate of change is not 
particularly rapid. Such areas may well be served by chapters in more wide- 
ranging annual reviews until the growth of the area warrants a separate annual 
review. 

Some annual reviews have been developed as commercial enterprises by private 
publishing concerns. Others, like the Annual Review, have been developed with 
subsidies from the Federal Government. Should annual reviews be subsidized, 
and, if so, for how long? Again, no clear answer can be given — certainly not 
from the present study. The Annual Review of Information Science and Technology 
was probably supported because the Federal Government has a substantial stake 
in more effective information rprocessing and professional communication, and 
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because the field was not commercially attractive to any publisher of annual 
reviews. Most of the funds provided by Government agencies to new publica- 
tions are regarded as "seed money," and are proffered with the more or less 
explicit e}q>ectation that the publication will become self-supporting in a. few 
years • The present study certainly confirms the wisdom of helping to initiate 
the Annual Review: the series has stimulated and will undoubtedly continue to 

stimulate professional communication and will thus help to accelerate technical 
progress, to the benefit of the Federal Government and the information com- 
munity as a whole. The study also raises some interesting questions about the 
effect of pressure toward self-support. Such pressure is obviously necessary: 
there are more potential annual reviews and other publications than could 
easily be supported with Federal funds, and eventual self-support must be 
encouraged. Yet, if too precipitous a rate of movement toward self-support 
is demanded, it can be achieved only by the adoption of qualityrdiluting short- 
cuts in chapter preparation, technical criticism and editing, or bibliographic 
accuracy, or— as indicated earlier— by moving toward a popularized or tutorial 
orientation. If such an orientation destroys the usefulness of an annual 
review for precii^ely the group that exercises technical influence and leader- 
ship in the field, then the pressure for rapid self-support is ultimately 
self-defeating to the sponsoring organization, whether it be a Federal agency, 
a professional society, or any other organization that wishes to achieve 
progress, rather than profit. 

Whatever the support for particular annual review publications, it seems 
essential to seek, wherever possible, empirical data on both effectiveness and 
iiTjpact, to the end that better-informed judgments about current and promising 
new communication tools can be made. The present study has provided an initial 
foundation of empirical data on which, it is hoped, a more comprehensive 
understanding of review literature can be built. 
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APPENDIX A: QUESTIONNAIRES AND COVER LETTERS 
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Annual Review Professional Use Study 



1 . How did you obtain the Annual Review copy 

____ purchased a personal copy 

__ used a desk or office reference copy 
purchased with company or staff funds 

borrowed it from an individual 



you have used? 

borrowed it from a university library 

boiTOwed it from a company library 
other (please specify) 



2 . 



How 



did you first hear of the Annual Review? 

preliminary announcement of contract 
award 

announcement or program prior to I966 
ADI Azinual Meeting 

1966 ADI Annual Meeting 

Journal advertisement 

pubMsher's direct mail advertisement 



(check one) 

__ colleague 
_ book review 

_ citation in a bibliography 
__ library 
_ bookstore 

__ other (please specify) ____ 
can't remember 



3 * Even if you have not read or skimmed all. chapters of the Annual Review, which omes 
do you consider to be within your particular area of professional interest? 

____ Professional Aspects of Information Science and Technology 

Information Needs and Uses in Science and Technology 

Content Analysis, -Specification and Control 

File Organization and Search Techniques 

Automated Language Processing 

Evaluation of Indexing Systems 

__ New Hardware Developments 

_____ Man-Machine Communication 

____ Information System Applications 

_____ Library Automation 

____ Information Centers and Services 

National Information Issues and Trends 



4 . Of the above 12 chapters, how many have you: read in detail? skimmed? 

5. About how many of the publications cited in the chapters within your particular areas 
of interest had you read before you saw them cited in the Annual Review? 

_____ practically none ____ some quite a few most 

6 . Did you examine any of these publications again as a result of reading the Annual 
Review? 

Yes: About how many did you examine again? 

1-5 11-15 21-30 

6-10 ____ i£- 20 _____ more than 30 

No 
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7. 



If you have examined any of the publications again, what were your reasons for 
examining them? To: 



____ review details you'd forgotten 

re-evaluate a publication in light of an Annual Review author's comments 

cite a publication or use the information in a report, review, or article of 

your own 

__ obtain information on other works by a publication's author 






SpGCiiy / 



8 . 



After reading the Annual Review, have you tried to obtain some of the publications 
cited in it that you hadn't read before? 



Yes: About how many? 

1-5 

6-10 



11-15 

16-20 



21-30 

more than 30 



No 



As a result of reading the Annual Review, have you tried to contact any of the authors 
mentioned for information regarding their current work? 



_____ Yes: About how many have you tried to contact? 

No 



10. Hcs the Annual Review itself ever suggested to you specific ideas for future research 
projects or studies? 



Yes 

No 



11 . 



V/hen did you last use the Annual Review? 
___ within the past week 
within the past month 



_____ within the past 3 months 
more than 3 months ago 

For what purpose did you use the Annual Review at that time? 



12 . 



Have you obtained from the Annual Review itself information about any of the following? 

ideas or theories a system 

methods or procedures a project 

results cr data other (please specify) 



1 

1 



an individual and his work 
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13* In general, how useful has the Annual Review been to you in serving each of the 
following goals? 



Have Not Actually 

Very Moderately Of Little Used It for This 

Useful Useful or No Use Function Yet 



Keeping up with current work in your 
own areas of interest 

Keeping up with current work in 
peripheral areas of Interest 

Checking on particular projects or 
ideas 

Checking on activities of individuals 

Learning about an area not within 
your professional specialty 

Reading the original literature more 
selectively 

Identifying areas of information sci- 
ence that require further research 

Allocating research and development 
funds more effectively 

Other (please specify) 



14. How often have you used the information you've obtained fr*an the Annual Review for 
each of the following activities? 

Often Sometimes Never 

Ordering documents or publications 

Searching the literature as a service to others 

Writing reports, articles and papers 
Preparing lectures or speeches 
Writing your own review of the literature 
Preparing a bibliography or reading list 
Exchanging information with your colleagues 
Other (please specify) 
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15. How important is each of the following information sources in helping to provide 
information you need for your current work? 



Very 

Important 



Do Not Use 
Of Little This 

Moderately or No Informetio 
Important Importance Source 



Colleagues within your own place of 
employment 

Colleagues outside your own place of 
employment 

Professional meetings and symposia 

Journal and periodical literature 

Technical reports 

Books, textbooks 

The Annual Review 

Preprints of periodicixl PjrbJ.cles 

Reprints of periodical articles 

Abstracting/indexing services 

Bibliographies 

Other (please specify) 



16. If one chapter of the Annual Review were to be published each month instead of collect- 
ing the chapters in book form annually, how do you think the value of the Annual 

Review would change? A different chapter each month would be? (check one) 

much less ^somewhat less ^equally ^somewhat more much more 

useful useful useful useful useful 

Why? 



17. In the future, what aspects of the Annual Review might be changed to make it more 
useful? 
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18. To which of these professional organizations or associations in the field of infor- 
mation science do you belong? 



ADI 


AMTCL 


____ ACS, Chem. Lit. Div. 


SLA 


TBRP! 


APA 


AIA 


AAAS 


Other (please specify) 


ACM 


STWP 





19. Which Jouroals and other scientific periodicals do you read or scan regularly (that 
is, almost every issue)? 



20. Approximately how many of each of the following materials have you authored or 
co-authored in the past five years? 

3ooks Technical reports 

Chapters in books Unpublished papers 

Journal (or periodical) articles Book reviews 

21. How many local or national professional meetings (conventions, conferences, symposia, 

etc. ) have you attended during the past year? At how many of these meetings 

did you present a paper? ____ 



22. How wwy years have you worked in the field of information science— i.e., documenta- 
tiai, library science, infomation systems research, or related fields? 

___ 20-30 years 

over 30 years 



23. 



___ IO-15 years 
15-20 years 



24 . 



less than 5 years 
____ 5-10 years 

In what type of organization are you currently employed? 

university ___ private foundation 

____ government agency ____ public library 

industrial or business firm ___ public school 

non-profit corporation or ____ other (pleetse specify) 

organization 



Please rank the following Job activities by placing a 1 in front of your most time- 
consuming activity, a 2 in front of your second most time-consuming activity, etc. 
Ignore activities you do not usually engage in. 



librazy reference or bibliogra^ic service 

other library service (e.g., cataloging, 
acquisitions) 

«search 

design or development 

manageznent or administration 

__ parogrammlng 

consulting in your professional specialty 



teaching 

studying for a degree 
. writing or editing 

publishing 

____ indexing or abstracting 
sales 
other 



er|c 



i. 
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25. 



Please describe area of your professional specialty for the most time- 
consuming activity you listed in question 24 (e.g., library science, library systems 
information systems, language processing, engineering, etc.) 



If you -have a different specialty for the second most time-consuming activity you 
listed, describe it too. ! — . 



26. What is your highest earned degree? 

no degree 

S. A. , S. S. 

D. L. S. 



M.A. , M.S. 

M.L.S. 

Ph.D., M.D., Ed.D. 
Other (please specify) 



Year earned: 

before 1935 

between 1935-19^0 
between 1940-1945 
between 1945-1950 



between 1950-1955 
between 1955-1960 
between i960- 1965 
after 1965 



27. Major field in which highest degree was earned: 



Library Science 

Psychology 

Sociology 

Linguistics 

Mathematics 



Computer Science 
Riysics 
Chemistry 
Engineering 
Biological Sciences 



Education 
Bus. Admin. 

English 

History 

Other (please specify 



28. Your sex: Male 



Female 



Please check here if you would like 'oo receive a summary report of the study 
findings. 
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QUESTIONNAIRE FOR SEEN/HEARD GROUP 
Annual Review Prores signal Use Study 



1. How did you first hear of the Annual Review? (check one) 

preliminary announcement of contract award 
announcement or program prior to 1966 ADI Annual Meeting 

1966 ADI Annual Meeting 

journal adverti.sement 

publisher's direct mail advertisement 

colleague 

book review 

citation in a bibliography 

library 

bookstore 

other (please specify) 

can't remember 



2 . The followirig is a list of Annual Review chapter topics. Please check the ones 
you consider to be within your particular area of professional interest. 

Educational Programs and Trends in Information Science 
Behavioral Studies of Information Needs and Uses 
Analysis, Description and Indexing of Document Content 
File Organization and Search Techniques 

Automated Language Processing and Computational Linguistics 

Evaluation Studies of Indexing Systems and Terminology 

■ New Hardware Developments for Information Systems 

Man-Machine Communication and Interaction Studies 

Information System Applications in Medicine, the Military, Law, Business, 
Chemistry/, and Education 

Library Automation 

Information Centers and Services 

r - 

NatiOxnal Information Issues and Trends 
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3. In the future, for which purposes do you think you may use the Annual Review? 

____ Keeping up with current work in your own areas of interest 

____ Keeping up with current work in peripheral areas of interest 

____ Checking on particular projects, ideas, or activities of individuals 

Other (please specify) 



U. How important is each of the following information sources in helping to provide 
information you need for your current work? 



Do Not Use 
Of Little This 

Very Moderately or No Information 
Important Important Importance Source 

Colleagues within your own place 

of employment — . — — n . 

Colleagues outside your own place 

of employment - — 

Professional meetings and symposia 

Journal and periodical literature 

Technical reports ____ __ 

Books, textbooks 

Preprints of periodical articles _____ 

Reprints of periodical articles 

Abstracting/indexing services _____ 

Bibliographies ____ 

Other (please specify) 



To which of these professional organizations or associations in the field of infor- 
mation science do you belong? 



ADI 


AMTCL 


ACS, Chem. Lit. Div. 


SLA 


1 KhiK 


APA 


ALA 


AAAS 


Other (please specify) 


ACM 


STWP 





O 



a 
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6. Which journals and other scientific periodicals do you read or scan regularly (that 
is, almost every issue)? 



7. Approximately how many of each of the following materials have you authored or 
co-authored in the past five years? 

Books Technical reports ____ 

Chapters in books Unpublished papers 

Journal (or periodical) articles Book reviews 



8. How many local or national professional meetings (conventions, conferences, symposia, 

etc.) have you attended during the past year ? At how many of these meetings 

did you present a paper? 



9. How many years have you worked in the field of information science— i.e. , documenta- 
tion, library science, information systems research, or related fields? 

less than 5 years IO-15 years 20-30 years 

5-10 years 15-20 years over 30 years 



10 . In what type of organization are you currently employed? 

university private foundation 

government agency ____ public library 

Industrial or business firm public school 



non-profit corporation or 
organization 



other (please specify) 



11 . Please rank the following job activities by placing a 1 in front of your most time- 
consuming activity, a 2 in front of your second most time -consuming activity, etc. 
Ignore activities you do not usually engage in. 



library reference or bibliographic service 

other librar;-r service (e.g., cataloging, 
acquisitions ) 

research 

design or development 
management or administration 
programming 

consulting in your professional specialty 



teaching 

studying for a degree 

writing or editing 

publishing 

indexing or abstracting 

sales 

other 



15 November I968 



104 



IIM-4125 



- 4 - 



) 



12 . Please describe the area of your professional specialty for the most time- 

consuming activity you listed in question 11 above (e.g., library science, library 
systems, information systems, language processing, engineering, etc.) 



If you have a different specialty for the second most time-consuming activity you 
listed, describe it too. 



13. What is your hipest earned degree? 

no degree 

S. A. , B. S. 

B. 1. S. 



M.A. , M.S. 
M.L.S. 



Ph.D. , M.D., Ed.D. 
Other (please specify) 



Year earned: 

before 1935 

_____ between 1935-1940 
_____ between 1940-1945 
between 1945-1950 



between 1950-1955 
between 1955-19^0 
between I96O-I965 
after I965 



l 4 . Major field in which highest degree was earned: 



Library Science 

_____ Psychology 
_____ Sociology 

Linguistics 

Mathematics 



Conputer Science 
Physics 
_____ Chemistiy 

Engineering 

Biological Sciences 



Edw-cation 
Bus,, .Admin. 

English 

History 

Other (please specify) 



15. Your sex: 



Male 



.Female 



Please check here if you would like to receive .a summary'' report of the study 
findings . 
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questionnaire for not-seen/heabd gboup 



Annual Review Professional Use Study 



1. How important is each of the following information sources in helping to provide 
inform/ation you need for your current work? 



Of Uttle 

Very Moderately or No 
Important Important Importance 



Do Not Use 
This 

Information 

Source 



Colleagues within your own place 
of employment 

Colleagues outside your own place 
of employment 

Professional meetings and symposia 
Journal and periodical literature 
Technical reports 
Books, textbooks 

Preprints of periodical articles 
Reprints of periodical articles 
Abstracting/indexing services 
Bibliographies 

Other (please specify) 



To which 


of these professional organizations 


or associations in the field 


mation science do you belong? 




ADI 


AMTCL 


ACS, Chem. Lit. Div. 


SLA 


lEIEE 


APA 


ALA 


AAAS 


Other (please specify) 


ACM 


STWP 
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3. Which journals and other scientific periodicals do you read or scan regularly (that 
is, almost every issue)? 



4. Approximately how many of each of the following materials have you authored or 
co-authored in the past five years? 

£QQjjg Technical reports 

i 'm Unpublished papers 

Journal (or periodical) articles Book reviews 



5. How many local or national professional meetings (conventions, conferences, symposia, 

etc.) have you attended during the past year ? At how many of these meetings 

did you f>resent a paper? 



6. How many years have you worked in the field of information science— i.e., documenta- 
tion, library science, information systems research, or related fields? 

less than 5 years IO-15 years 20-30 years 

5-10 years 15-20 years over 30 years 



7. 



In what type of organization are you 
university 

government agency 

industrial or business firm 

non-profit corporation or 

organization 



currently employed? 

privEite foundation 

public library 

other (please specify) 



8 . 



Please rank the following job activities by placing a 1 in front of your most time- 
consuming activity, a 2 in front of your second most time-consuming activity, etc. 
Ignore activities you do not usually engage in. 



library reference or bibliographic service 

other library service (e.g., cataloging, 
acquisitions) 

research 

design or development 
management or administration 
programming 

consulting in your professional specialty 



studying for a degree' 
writing or editing 
publishing 

indexing or abstracting 

sales 

other 
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9. Please describe the area of your professional specialty for the most time- 

consuming activity you listed in question 8 above (e.g.^ library science^ library 
systems, information systems, language processing, engineering, etc.) 



If you have a different specialty for .the second most time-consuming activity you 
listed, describe it too. 



LO. What is your highest earned degree? 
no degree 



n.A. 



P.R 



B.Ii. S. 

Year earned; 

before 1935 

between 1935-1940 

between 1940-1945 

between 1945-1950 



M.A., M.S. 

M.L.S. 

Ph.D. , M.D., Ed.D. 

Other (please specify) 

between 1950-1955 
between 1955-19^0 
between I96O-I965 
after 1965 



11. Major field in which 

Library Science 

Psychology 

Sociology 

Linguistics 

Mathematics 



degree was earned: 
Ccmputer Science 

Physics 

Chemistry 

Engineering 
Biological Sciences 



Education 
Bus. Admin. 

English 

History 

Other (please specif^) 



12. Your sex: Male Female 

Please check here if you would like to receive a summary report of the study 
findings. 
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Institute for CoDimunlcation Research 
Stanford University 
Stanford^ California 

September 13, 19^7 



Dear Colleague: 

Recently you were kind enough to reply to a postcard regarding 
use of the Annual Review of Information Science end Technolo^ » 
Stanford University’s Institute for Ccxnmunicetion Research and 
System Development Corporation are currently examining the impact 
of the Annual Review as an information tool, and we would like to 
have the benefit of your experience and views for this study. 

Specifically, we woi\ld appreciate your completing this brief 
questionnaire and returning it to the Institute for Communication 
Research in the envelope provided. The identification number on 
the questionnaire is to permit us to contact non-respondents, in 
accordance with standard sampling procedures. The information 
you provide will not be associated in any way with your name 
during data analysis. 

We will be pleased to send you a summary report of study findings 
if you indicate your interest it the end of the questionnaire. 

Thank you for your cooperation. 



Linda Rarris 
Study Director 



COVER LETTER TO USER 
AND seen/heard GROUPS 
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Institute for Cooununlcatlon Besearch 
Stanford University 
Stanford, California 

September 25, 1967 



Dear Colleague: 

Becently you were Ir^-id enough to reply to a postcard regarding 
use of the Annual Review of Information Science and Technology . 
Stanford University's Institute for Canmiuiication Besearch and 
System Development Corporation, are currently examining the impact 
of the Annual Review as an information tool, and part of this 
study involves analysis of information sources of those who have 
not yet read the Annual Review. Thus we would like to have the 
benefit of your participation in this study. 

Specifically, we would appreciate your completing this brief 
questionnaire and returning it to the Institute for Communication 
Research in the envelope provided. The identification number on 
the questionnaire is to permit us to contact non-repondents, in 
accordance with standard sampling procedures. The information 
you provide will not be associated in any way with your name 
during data analysis. 

We will be pleased to send you a summary report of study findings 
if you indicate your interest at the end of the questionnaire. 

Thank you for your cooperation. 



Linda Harris 
Study Director 



COVER LETTER TO 

not-seen/he:ard group 
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Institute for Ccxnraunication Research 
Stanford University 
Stanford, California 

October 'J, I967 



Dear Colleague: 

Recently, we sent you a brief questionnaire regarding your use of the 
Annual Review of Infonnation Science and Technology « Since, we have 
not yet received a reply, I am writing to you again, to ask that you 
complete the questionnaire at your earliest convenience. . ’ 

As you are aware, our study on information use patterns in the 
information science field depends upon successful collection of data 
from a representative sample of professionals. I am especially eager 
for your views to be included in our study, because you have been a 
user of the first volume of the Annual Review. Your reactions to it 
will therefore help us to assess its impact as an information tool. 

In case the previous questionnaire sent to you was lost in the mail 
or misplaced, another copy is enclosed,, together wi-to a reply envelope. 
(If you have returned a questionnaire to use by the time this letter 
reaches you, please disregard it. ) The identification number on the 
questionnaire is to permit us to contact non-respondents, and also 
to guarantee that the information you provide will not be associated 
in any way with your name during data analysis. 

We will be pleased to send you a summary report of study findings if 
you indicate your interest at the end of the questionnaire. 

Thank you for your cooperation. 



Sincerely yours. 



Linda Harris 
Study Director 



FOLLOW-UP LETTER 
TO USERS 
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Institute for Ccmmunication Research 
Stanford University 
Stanford, California 

October 17, 1967 



Dear Colleague: 

Recently, we sent you a brief questionnaire regarding your profes- 
sional information sources and your contact with the Annual Review 
of Information Science and Technology * Since we have not yet 
received a reply, I am writing to you again, to ask that you complete 
the questionnaire at your earliest convenience. 

As you are aware, our study on information use patterns in the 
information science field depends upon successful collection of 
data from a representative sample of professionals. I am eager 
for your views to be included in our study. Although you have not 
been a user of the Annual Review, the information we have requested 
from you will help us to assess its Impact as an information tool. 

In case the previous questionnaire sent to you was lost in the mail 
or misplaced, another copy is enclosed, together with a reply 
envelope, (if you have returned a questionnaire to us by the 
time this letter reaches you, please disregard it.) The identi- 
fication number on the questionnaire is to permit us to contact 
non-respondents, and also to gu6u:antee that the information you 
provide will not be associated in any way with your name during 
data analysis. 

We will be pleased to send you a summary report of study findings 
if you indicate your interest at the end of the questionnaire. 

Thank you for your cooperation. 



Sincerely yours. 



Linda Harris 
Study Director 
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Institute for Communication Research 
Stanford University 
Stanford, California 

October 19, 19^7 



Dear Colleague: 

Recently, we sent you a brief questionnaire regarding your 
professional information sources. Since we have not yet 
received a reply, I am writing to you again, to ask that you 
complete the questionnaire at. your earliest convenience. 

As you are aware, our study on information use patterns in the 
information science field depends upon successful collection 
of data from a representative sample of professionals. We are 
eager for your views to be included in our study, even though 
you are not yet a user of the Annual Review of Information 
Scie nce and Technology , because knowledge of other information 
sources relevant to our field will assist our analysis. 

In case the previous .questionnaire sent to yo\x was lost in the 
mail or misplaced, another copy is enclosed, together with a 
reply envelope. (If you have returned a questionnaire to us 
by the time this letter reaches you, please disregard it.) 

Ihe identification number on the questionnaire is to permit 
UiS to .contact non-respondents , and also to guarantee that the 
information you provide will not be associated in any way 
with your name during data analysis. 

We will be pleased to send you a summary report of study findings 
if you indicate your interest at the end of the questionnaire. 

Thank you for your cooperation. 

Sincerely yours. 



Linda Harris 
Study Director 
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SUMMARY REPORT 



STANFORD UNIVERSITY 



STANFORD, CALIFORNIA 94305 



.INSTITUTE FOR COMMUNICATION RESEARCH 



CYPRESS HALL 
Telephone: 
415/321-2300 
Extension 2753 



October 14, 1968 



Dear Colleague in Information Science: 

At this time last year, you were answering a questionnaire about your use of 
Volume 1 of the Annual Review of Information Science and Technology (ARIST) . 

You indicated that you would like to receive a summary of the results of the 
survey, so I am writing to you to report some major findings and conclusions. 

A more detailed report of study findings will be submitted to the U. S. Office 
of Education after November 15. Copies of the report are likely to be avail- 
able through the ERIC system. 

Thanks to your participation, the survey was very successful. The final data 
set contains 2012 questionnaires from readers and nonreaders of the ARIST, and 
on possible ways to improve it in the future. 

Questions covered the extent and purposes of use of the book, the information- 
seeking behavior it prompted (such as "writing to an author for more infor- 
mation") , important information sources used in addition to the ARIST, and 
demographic information. We found that respondents read, on the average, four 
chapters of the ARIST, primarily to keep up with the literature in their own 
areas of professional interest. Other major uses of the book, however, are for 
reference, for learning about a peripheral area of interest, and for teaching 
or lecturing. 

ARIST readers tend to be more active professionally than nonreaders: they 

attend meetings and symposia, present papers at these meetings, belong to 
several professional associations, write journal articles and technical reports, 
and read several journals on a regular basis. These results and others are 
presented in more detail on the following pages. 

By now, you probably have seen Volumes 2 and (possibly) 3 of the ARIST, and 
your evaluation of the book's value may have changed since you answered our 
questionnaire. An informal group discussion with the editorial staff and 
Advisory Committee members is scheduled for October 21, 1968 at the ASIS 
Meeting in Columbus. It would be a pleasure to see you there. 

Once again, let me express my appreciation for your participation in the study. 



Cordially 




Linda Harris 
Study Director 
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SUMMARY OF IMPACT SOUDY OF THE 
ANNUAL REVIEW OF INFORMATION SCIENCE AND TECHNDIiOGfY* 

Linda Harris 

Institute for Communication Research, 

Stanford University 



The Annual Review of Information Science and Technology (ARIST) was i '■roduced 
as a critical review of current literature to aid professionals in the difficult 
and often frustrating task of keeping up with the literature in their special- 
ties. The purpose of this survey was to determine how well the ARIST has 
accomplished this goal, and how the book has affected the process and patterns 
of scientific communication. To obtain a valid measure of those patterns, we 
compared readers and nonreaders on several measures of information-processing 
activities and demographic attributes. 

Our first task was to identify readers and nonreaders of the ARIST within the 
universe of information scientists. Business reply postcards were sent to the 
10,000 members of four professional organizations: American Society for Infor- 

mation Scientists (A^IS-rfomerly American Documentation Institute), Special 
Libraries Association, Special Interest Group on Information Retrieval of the 
Association for Computing Machinery, and Association for Computational Linguis- 
tics (formerly Association for Machine Translation and Corq^utational Linguis- 
tics). 

About 3,700 postcards were returned, almost equally divided among readers, non- 
readers, and a middle group who had seen the ARIST or heard of it, but not yet 
read it carefully. Detailed questionnaires were sent to all of the readers; a 
separate version excluding questions about readership of the ARIST was sent to 
the other two subgroups. After two mailings, the response rate was 70^, or 
2,012 questionnaires. 

The two major portions of the analysis were a comparison of the three respondent 
subgroups on demographic and information-processing attributes, and a separate 
analysis of the readers' uses of the book. 

The readers, nonreaders, and "seen -heard" subgroups differed very little on 
personal attributes such as number of years in information science, highest 
earned degree, major field for that degree, and place of employment. A typical 
respondent from any of these subgroups probably would have a "professional age" 



* This study was partially supported by a grant from the U.S, Office of Educa- 
tion to System Development Corporation (SDC), with Dr. Carlos A, Cuadra as 
Principal Investigator. SDC cosixDnsored the study, with assistance from 
Encyclopaedia Britannica, publisher of the series. 
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of less than 10 years in information science and would be an en 5 )loyee of an 
industrial or business firm. He is also likely to hold a degree, at either the 
Bachelor or Masters level, in library science. 

However, the groups do differ considerably in several other characteristics. 

For example, almost 80^ of the readers are ASIS members, whereas only 45% of 
the seen-heard subgroup and 19% of the nonreaders belong to ASIS. (This shows 
that the ARIST is publicized mainly through ASIS channels.) In addition, the 
highest percentages of respondents in all three subgroups are employed in 
management, design, and research, as opposed to library and other activities. 
Library activities are not well represented among the reader group, perhaps 
because the librarians tend to skim the book for awareness of content and not 
for absorption of information. 

These demographic attributes do not fully account for differences in readership. 
Other factors--such as professional communication patterns — were also investi- 
gated, Previous research has shown that certain patterns of information 
exchange and levels of communicative activity are characteristic of the scien- 
tists who are at the forefront of developments in their disciplines. Our data 
also support these findings. ARIST readers use information channels more than 
respondents in either of the other two subgroups. The typical reader is a 
prolific author of papers and technical articles; he reads almost eveiy issue 
of at least five professional journals; he attends about three professional 
meetings each year, and he presents a paper for at least one of these meetings. 
One conclusion from these results is that the ARIST apparently does not serve 
as a substitute for professional journals, because journal readership and ARIST 
readership are highly correlated. Thus, we find that the ARIST functions as an 
addition to a list of information sources used by already-active information 
scientists. 

What goals of information-seeking does the ARIST satisfy, and how does it com- 
pare with other information sources used by professionals in the field? 

The data suggest several functions performed by the ARIST that might be labeled 
"continuing education." One manifestation of continuing education is current 
awareness of the literature. Almost 80% of the readers use the ARIST to keep 
up in their own areas of interest, and 74% also find it useful to keep up in 
peripheral areas. To many readers, "keeping up" seems to mean "making sure I 
haven't missed in^jortant literature in my field during the past year." 

Other irrrportant aspects of continuing education are orientation, interpretation, 
and critique. Over half the readers use the ARIST to learn about an area out- 
side their specialties. Furthermore, almost 40% of the readers use it to read 
the primary literature more selectively, and more than half report that they 
reexamined articles after having read of them in the ARIST. The importance 
assigned by readers to such critique (which is not widely available from 
journals) is reiterated by numerous suggestions for more critique in future 
volumes of the ARIST. 



15 November 1968 



120 



TM-4125 



3 



The reference aspect of information-seeking— checking facts, references, and 
conclusions— is also supported by the survey data. Over half the readers find 
the ARIST useful for checking on particular projects or ideas, and 22^ use it 
to check on activities of individuals. 

Another interesting but rather mysterious finding is that readers who have been 
information scientists for less than 10 years do not use the ARIST as much as 
those who have been in the field for more than 10 years. The newcomers read 
the ARIST much more for orientation than they do for current awareness. Neither 
demographic nor information-use attributes fully explain this relationship. 

For many readers, all these uses of the ARIST are intermediate steps to other 
information goals. For exaii 5 )le, a number of the respondents who teach univer- 
sity courses use the book to introduce their students to information science 
and to guide them through the literature. 

Despite the wide range of functions reportedly performed by the .ARIST, no sig- 
nificant correlation was found between the ' purpose of ARIST use and the impor- 
tance attributed to other information sources, such as colleagues, journals, 
professional meetings, reprints of articles, etc. This is surprising, because 
other data in our study confirm the relationship between, for example, journal 
readership and ARIST readership, and we conclude that some response bias may be 
operating here to prevent the painting of a complete statistical picture of 
information use patterns. 

Although certain information use patterns are not clear, the data do allow us 
to document information-seeking behavior that preceded and followed use of the 
ARIST. Readers were asked how much contact they had had with the cited publi- 
cations before reading the ARIST. Over 60^ had had at least some contact with 
cited publications within their areas of interest, and 38 ^ were well acquainted 
with many of the publications. 

Respondents who had had relatively little prior contact with the cited publica- 
tions tend to have read comparatively little in the ARIST, but those with more 
previous contact are equally likely to report high ARIST readership as they are 
to report low readership. Readers with more previous contact with the cited 
publications tend to find the ARIST less useful for current awareness than for 
other p\irposes. Although we do not know the exact function that previous con- 
tact with cited publications performs, we know that it does not act as a 
deterrent to use of the ARIST. This variable deserves further study; it could 
become a valuable predictor of readership in other contexts, such as the initia- 
tion of new information services. 

Information-seeking after reading the ARIST was also examined in the study. We 
have already mentioned the fact that 53^^ of the readers were led to reexamine 
publications cited in the ARIST. In addition, more than half of the readers 
tried to obtain some of the publications cited in the ARIST that they had not 
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read before, and 10^ tried to contact an author who was cited. Finally, 
state that the ARIST has suggested to them specific ideas for future research 
projects. All of these findings are positively correlated with extent of 
readership. As number of chapters read increases, so does post-reading behavior 
for all of these measures. 

In conclusion, the data have helped to provide valuable insights into use of the 
ARIST. Such variables as previous contact with cited publications, number of 
years in information science, and browsing as a purpose for use of the ARIST 
can help to provide a basis for the planning and funding of annual review publi- 
cations. The data collected in this initial study should also provide a founda- 
tion for future studies in this area. 



